Next Article in Journal
Using Game Design to Teach Informatics and Society Topics in Secondary Schools
Previous Article in Journal
A Survey to Understand Emotional Situations on the Road and What They Mean for Affective Automotive UIs
Previous Article in Special Issue
Multimodal Technologies in LEGO House: A Social Semiotic Perspective
Article Menu

Export Article

Multimodal Technologies Interact. 2018, 2(4), 76;

“There Was No Green Tick”: Discovering the Functions of a Widget in a Joint Problem-Solving Activity and the Consequences for the Participants’ Discovering Process
University of Luxembourg (UL), L-4366 Esch-sur-Alzette, Luxembourg
Luxembourg Institute of Science and Technology (LIST), L-4362 Esch-sur-Alzette, Luxembourg
Author to whom correspondence should be addressed.
Received: 13 August 2018 / Accepted: 9 October 2018 / Published: 26 October 2018


In recent years, tangible user interfaces (TUI) have gained in popularity in educational contexts, among others to implement problem-solving and discovery learning science activities. In the context of an interdisciplinary and cross-institutional collaboration, we conducted a multimodal EMCA-based video user study involving a TUI-mediated bicycle mechanics simulation. This article focusses on the discovering work of a group of three students with regard to a particular tangible object (a red button), designed to support participants engagement with the underlying physics aspects and its consequences with regard to their engagement with the targeted mechanics aspects.
discovery work; joint activity; tangible user interface (TUI); multimodal conversation analysis

1. Introduction

In the past decades computers in all its versions including mobile devices, smartboards, multi-touch boards and tangible system have become omnipresent in educational settings. Among them tangible interfaces (TUI) are gaining in popularity. Due to their “inherent ability to interplay between the physical and digital domains” they are said to be very intuitive and so have “gathered interest for applications in numerous domains” [1] (p. 215). One is simulations in scientific domains which enable participants to manipulate objects or parameters leading to dynamic system-generated responses which are “based on a set of underlying rules, models, or computations” [2] (p. 729). So, participants can perform experiments and (re-)discover the properties of the underlying model (e.g., a law of physics) [3]. For example, we showed elsewhere [4] how two students, while engaged in a problem-solving physics activity, discovered in situ that a specific factor influences one of the target values.
This problem-solving activity [4], which is also investigated here, is an instance of a simulation of a biker’s applied force and work done to the bike in a changeable landscape. Participants are required to repeatedly change the gears, the inclination and coarseness of the ground, as well as the distance of the trip in order to explore and discover how these adjustments impact the force and work. In this process, they are guided by a series of tasks with an increasing difficulty, each requiring the parameters to be set on different values. This and other applications were developed by teachers who participated in the ERASMUS+ project “Re-Engage” [5] led by the Luxembourg Institute of Science and Technology (LIST). In the context of a commencing interdisciplinary and cross-institutional collaboration (LIST and UL), we saw the opportunity to conduct an ethnomethodological/conversation analytic (EMCA) informed video-based qualitative user study with three small groups of adult students. The above-mentioned bike mechanics scenario appeared to be the most appealing to us to investigate how participants engage in TUI-mediated joint problem-solving activities and to explore the implementability of the developed prototypes in educational contexts. When we first looked at the footage, we found that two groups had efficiently resolved the problem by relying on the tangible user interface. However, when we looked closer at the video data, we noticed that, from a certain moment on, both groups speeded up the resolution of the proposed assignments by relying on a specific tangible object (a red button). They did so, however, in a way that went against the intentions of the designers, since the participants no longer engaged with the targeted physics or bicycle mechanics aspect of the activity. Indeed, the red button widget (see Section 2.3.) was introduced into the design of the activity to provide students with an opportunity to discuss and to reflect their way of setting the input variables before checking their impact on the output variables (by pressing the button). Our detailed analysis will show, that this did work out reasonably well during a certain phase of the activity but only as long as they had not uncovered all the features of the red button.
Besides teaching us some lessons about designing TUI-mediated learning activities, the moment-by-moment EMCA-based multimodal analysis [6,7,8,9] of the data also provided us with a valuable opportunity to study in detail participants’ joint discovery work [4,10,11]. This in a twofold and intertwined way: Firstly, since we had left the participants in the dark about the features of the red button widget, they had to and did engage in a situated discovering process of jointly (re-)constructing its embedded functions. Secondly, they integrated their corresponding findings in the problem-solving activity not only to discover some aspects of the underlying bicycle mechanics model but also upon uncovering the last function, to exhaust the possibilities of the underlying mechanisms of the interface in order to finish the remaining assignments in a minimum of time.
In this paper, we rely on the video data of the first group to highlight how the participants are doing discovering. We do this in order to contribute to a better understanding of how discoveries are produced within a context of digitally mediated joint activities and to gain design knowledge with regard to supporting the triggering of episodes in which participants engage in joint reflections and discoveries. Before we get to the account of our analysis, we shall first describe the development of the corresponding TUI-mediated activity (Section 2), provide some reflections on discovery learning and discovery work (Section 3), lay out our study design (Section 4), present our detailed analysis of six excerpts (Section 5), and, finally, draw our conclusions with regard to the organization of participants’ discovery work in TUI-mediated problem-solving activities and some related design issues (Section 6).

2. The TUI-Mediated Bicycle Problem-Solving Activity

2.1. Tangible User Interfaces

Shared interfaces such as multi-touch tables and tangible systems mediate and support collaboration by allowing co-located participants to simultaneously interact with digital information [12,13]. While multi-touch tabletop interfaces are operated using finger touches, tangible tabletop interfaces make use of tangible widgets to interact with the system. Here, the latter consist of a physical handle (a graspable object, participants manipulate to interact with the system) and a corona (a visual feedback element). This tangibility potentially facilitates the participants’ interaction with the system by building on everyday experiences of the physical environment, by enabling bi-manual control and by providing a “tight coupling of control of the physical object and the manipulation of its digital representation” [14] (p. 368). Furthermore, compared to multi-touch tabletops, tangible tabletop interfaces were found to better support socially oriented actions [15]. For instance, they allow the organization of personal, group and storage territories [16], enhance the visibility of group members’ interactions and promote equity of participation [17]. So, TUIs are particularly suited to serving as a platform for implementing joint problem-solving scenarios.

2.2. Original Context of the Bicycle Activity

The here investigated TUI-mediated bicycle problem solving activity deals with the setup of a bike and the planning of a trip (see Figure 1). Users can explore how different parameters (e.g., pedal gear, inclination) impact the force on pedal and the consumed energy (work). The corresponding idea was initiated in the context of the Erasmus+ project “Re-Engage” [5]. The later deals with the creation of tools and guides to support teachers in implementing new teaching and learning practices in their classrooms. One of the practices offered to the teachers is the use of tangible tabletop interfaces with small Microworlds based on structural equations. A Microworld is understood here as a computational environment that embodies or instantiates a mathematical or scientific subdomain; the central objects and relations of this subdomain are provided as an interactive representation, with the aim to make them accessible to new learners [18].
To support teachers in creating their own Microworlds, COPSE (Collaborative Problem Solving Environment) [19] was developed providing well-defined building blocks which can be combined and customized in various ways. In particular, COPSE is based on TULIP (Tangible User Interface Library) [20] and makes use of the notions of widgets (provide input and localized feedback), equations (define the model) and scenes (visualize feedback), which are specified in the form of structured text. While this framework provides some constraints on the different effects and features that can be implemented on the table, it considerably reduces the time and technical skills needed for an instantiation. In a workshop, teachers from different countries had worked in small groups to select a topic, identify variables, model equations, create graphics and widgets, and, with the help of a researcher, had generated a running version of six simulation environments (preparing a béchamel sauce, comparing transport means, how photometers work, setting up your bike, managing a budget, food categories) related to different science education domains.

2.3. Reworking the Bicycle Activity

From the elaborated scenarios, we retained the bicycle mechanics problem for our study, since it builds upon an often-misconceived concept, that is, that the force on pedal decreases when selecting a larger sized gear at the wheel but increases with a larger sized gear at the pedal. Furthermore, it fits into the curricula of both primary and secondary education and it is of interest for a larger public, for example, in the context of science fares.
In order to identify missing features and visualizations we made use of multidisciplinary design sessions and rapid prototyping. While the first version, as elaborated by the teachers, proposed the basic structure of the Microworld (including the widgets, equations and scenes), the focus of the refinements for the second version was also to enhance the pedagogical flavour. In this sense, we added a series of tasks and a red button as a checking device (see below and Figure 2). As soon as the ensuing modifications were implemented, we met again to test the new version and to refine it on the fly. In the following, we shall describe the basic structure of the Microworld, as well as the main refinements we did as part of our redesign. It is also the version the participants in our study worked with.
The redesigned Microworld implements the following five input variables (possible values in brackets): pedal gear (1–3); wheel gear (1–7); inclination of the ground (0%; 10%; 20%); coarseness of the ground (asphalt 0.2; sand 0.4); distance covered (0–30 km). The two output variables “force on pedal” and “work/energy” were defined via the corresponding equations (In this version, the inclination was, by mistake, interpreted as an angle whereas it was indicated to the users as percentage. Therefore, the values are much too high. In a more recent version this error has been corrected.) All the variables are associated with widgets (Figure 1a) which are composed of graspable physical objects of different shapes and colours (Figure 1b), hereinafter referred to as tangible, and visual feedback elements called coronas. The parameters of the ground (inclination, coarseness, distance) use round tangibles, with a three-dimensional icon on top. When placed on the table, a text box is visualized as a corona underneath the tangible to show the current value of that parameter. The pedal and wheel gear are represented through round tangibles with simple labels for both types of gears. In addition, a series of images of gears of different sizes are added, which each are bound to one of the values of the gear variables. By rotating the tangible to the right, the user changes the variable and eventually sees the size of the gear increasing and vice-versa. The manipulations of the inclination and ground widgets are also visualized in real time through seven background images: a light blue image with clouds (see Figure 2), as well as two different colors of ground (grey and beige) each with 3 different inclinations (0%, 10% and 20%), to be shown as a 2nd layer on top of the sky layer.
To turn the Microworld into a problem-solving activity we created a “rule induction problem,” that is, a problem that requires learners to “induce the rules governing how some system operates” [21] (p. 98). In order to achieve this, we created a series of tasks with increasing complexity. During the first task (“In the following you will have to solve a number of tasks. To start, explore the objects on the tabletop and observe what happens. When you feel ready, you may proceed to task 1.”), labelled “Start”, we wanted to provide the users with the opportunity to freely explore the Microworld to understand how to work with the tangibles and to get a first idea on how parameters impact each other. The subsequent tasks (N° 1–6) all deal with controlling the variables of the Microworld in order to achieve a given result. Users are either required to minimize or maximize parameters (e.g., “In which situation the biker consumes the most energy?”), or to achieve a certain range (“The biker wants to consume between 4100 and 4200 J. How can the biker achieve this?”) or even an exact number (“The biker wants to consume 8486 J and apply 2709 N. How can the biker achieve this?”). Our idea was that progressively, these outputs are harder to achieve, and users would be required to more fully understand the underlying rules in order to solve the task. In particular, they should realize that the force on pedal is impacted through the pedal and wheel gear, the inclination of the ground, and the coarseness of the ground, whereas the energy is impacted through the inclination of the ground, coarseness of the ground and the distance. Regarding the gears, the underlying rule is the force on pedal increases with a larger pedal gear but decreases with a larger wheel gear. At the end, the users are confronted with test questions (e.g., “What object(s) does/do not influence the force on pedal?”), requiring the users to take the appropriate tangibles off the table. To be able to iterate through the tasks, we added a widget (task wheel) whose handle needs to be rotated in order to move to the next task. The task description is shown as a corona next to that tangible and changes when the latter is rotated.
To make non-reflected trial and error strategies more difficult, we decided to add a particular widget (red button) to visualize the results of the users’ manipulations: when (and as long as) they physically press it, the current values of the independent variables (“force on pedal” and “work”) are displayed in the coronas of the output widgets. The underlying idea is that participants have the opportunity to take their time to jointly discuss a next setting, test it and reflect on the results in order to adjust the settings. We further provided users with feedback when the task is solved correctly. In this case, a green tick is added next to the task description, as well as a green round corona around the button (see Figure 2). This feedback is coupled to the button too and only shown when the button is pressed.

3. Discovery Learning and Discovery Work

The here designed TUI-mediated problem-solving activity incorporates the pedagogical strategy of scientific discovery learning where the main task of the learners “is to infer the characteristics of the model underlying the simulation” basically by “changing values of input variables and observing the resulting changes in values of output variable” [22] (p. 180). In other words, the science curricular discoverables embedded in the activity correspond to what Atkinson and Delamont [23] have termed “cold discoveries”, that is “the findings of previous scientific inquiries into settled matters” are “reenacted for pedagogical purposes” [11] (p. 41). In contrast, a hot discovery would be an inquiry into questions where there is no available answer yet [10,19]. The idea of discovery learning as a beneficial pedagogical approach has a long history [24,25] and the technological developments of the last decades come with the promise of increasing its efficiency [3]. However, research has pointed out that, overall, students very often have problems to identify variables or to generate hypotheses [3]. Due to this and other discrepant findings there is an “ongoing discussion about guided-versus-pure discovery learning” asking for “further studies on what and how to guide and scaffold in all forms of discovery learning” [26] (p. 1012).
One way that we attempted to guide the discovery process (from a pedagogical point of view) was the introduction of assignments/tasks with an increasing difficulty and self-revealing widgets. Hence, an aspect of our user study (not developed here) was also to investigate to what extend these measures worked and further guidance is needed to provide participants with a challenging but not frustrating learning experience. Contrary to our expectations the red button, a central widget in our design, turned out to be problematic in terms of self-explanation and ambiguous with regard to the accomplishment of the pedagogical task. Indeed, the participants of two groups, in a sense, “hacked” the red button. From a certain moment on (see Section 5.5.), they kept it pushed while continuously changing the input variables until the corona of the widget turns green. In a sense, they favoured a simple-quantified goal (the corona should turn green as quickly as possible) instead of a more abstract goal (grasping the fundamental laws explaining the simulation). This shows how difficult it is to design an activity where the intended pedagogical aims and ideas directly translate into the learners’ conduct and it calls for a research and design approach that focuses on “what actually happens, that is, on the task-in-process, rather than on what is intended to happen, that is, on the task-as-workplan” [27] (p. 95).
Discovery learning is generally conceived of as a constructivist approach to education in the sense of Piaget and the early Bruner [28]. This means that it was derived from studies in cognitive psychology and basically targets learners’ conceptual change which is traditionally being conceived of as being located in the brain [26]. Now, if discovery learning is a theory of instruction implying that things are learned more deeply when they are learned through discovery (as opposed to being told), then this raises the question of how the discovering actually gets done by the participants as an interactional matter. Since the meaning of a discovery is tied to the particulars of the situation in question, it is best studied within the specifics of that situation. From this perspective, we need to rely on a different methodology from one that involves collecting large samples, aggregating data and applying a statistical treatment leading to a general claim designed to apply to all situations. Instead, we are making a claim about how things got done in some specific occasion by some specific cohort. This is what is achieved by applying an ethnomethodological conversation analytic approach (EMCA) (see Section 4.2.).
Within this framework discovery work is conceived of as an interactional matter, during which participants display to one another that “something new enters the world held in common” by them and so we can witness “how the matter so treated transitions from a dawning possibility […] to an established thing” [29]. So, we do not investigate here discovery as a psychological phenomenon referring to a private event taking place in the realm of the mental. Rather we study another part that has to do with “how we display to others that we have to come to view the world in a different way” [29]. To differentiate this part from the psychological event of discovery and to emphasize that our research focus is on the interactional accomplishment of producing an intersubjectively ratified understanding, we shall rely here on the term “discovery work” [10,11,29,30].
To bring about this process (and others) as a coherent and intelligible course of action, participants organize their conduct not solely in talk but through the simultaneous use of multiple semiotic resources [31]. Thus, we relied on video recordings to document the social interactions among participants and their interactions with the TUI. So, in the following section we shall describe how we constructed and analyzed our video data.

4. Setting and Study Design

4.1. Constructing Video Data

To construct the video data for the study, we set up an experimentation room, having the tangible tabletop interface in the centre and the tangibles arranged on the tabletop. To capture participants’ interactions among them and with the TUI with great accuracy and detail, we recorded the activity from different perspectives. The recording equipment was composed of four fixed cameras and a separate sound recorder. The resulting sound and video data streams were connected within one space to generate an “expanded-around view” of the ongoing event (Figure 3); elsewhere we, we termed this apparatus “joint screen” [32]. Nevertheless, for reasons of convenience and to ensure that the chosen frame grabs in the transcripts are not too small to recognize relevant details, we chose here to rely on images from either one or two camera perspectives to support our analysis.
We recruited the study participants among our students and their acquaintances (opportunity sample) and retained seven participants aged 18–28. Prior to being recorded, participants were all provided with an information sheet about the study and its context, given the opportunity to ask questions and asked to fill in and sign a consent form. The here gathered sample size may appear small and, probably, would be so to build a collection. However, CA holds a methodological and epistemological position that has been termed “the method of instances” [33] (p. 50). This means that the primary task of CA is “the discovery, description and analysis of complex interactional phenomena as socially produced phenomena in their own right”; and that the occurrence of one such instance is enough to study its organization. The resulting claim is not one about the frequency of its re-occurrence but one of the cultural availability of the machinery for its production, involving participants’ competencies and therefore being “possibly (and probably) reproducible” [33] (p. 50).
During their time at grammar school, the participants had all attended physics classes covering the scientific concepts having a direct bearing on the resolution of the tasks (force, energy, work). However, they were not familiar with tangible tabletop interfaces. Participants were divided in three small groups and we let them decide how to do this, so that they would not have to become acquainted with one another during the activity. For each run, we set the widgets in the same place and to the same value and we set the task widget to the “start” position. Outside the room, we provided them with the necessary verbal instructions, which basically consisted in asking them to go inside the room, to read the task provided on the table and to solve it. Since we were also interested to what extend the designed problem situation would work as a stand-alone activity and to what extend the widgets afforded their designed usability, we provided no particular information on how to use the latter. During the problem-solving sessions, we stayed outside the experiment room to avoid that our presence impacts the participants’ interactions and strategies. In case a problem would occur, the participants were told to come outside the room and ask us for help.
Since we did not elaborate on the purposes of the widgets, they had not received any information regarding the functioning of the red button either, so they had to figure out on their own how to use it to support their problem-solving process. In addition, despite our expectations, the groups did not make use of the “start” task to truly explore the tool and to notice all the functions of the widgets. Therefore, many features and in particular those of the red button were still hidden when the participants began with task number 1. Participants only discovered its functions while progressing with the different tasks. Given these circumstances, our setting became a “perspicuous setting” [34], meaning that it was particularly well suited to study how matters discovered were managed by the study participants. To develop a better understanding on the participants’ discovering work with regard to these different features, we therefore focused our closer analysis on instances where they notice a new potential function of the red button and how they take it up to accomplish the task. We further point out how their “discoveries” have consequences upon their more curricular discovery work and their task accomplishment conduct.

4.2. Studying (Inter-)Action from an EMCA Perspective

To study the co-construction of meaning and the details of action as being sequentially arranged moment-by-moment by the participants, within the very context of their situated activity, we rely on an ethnomethodological conversation analytic (EMCA) [6,7,8] inspired approach. Over the past decades there has been a growing literature of EMCA-informed research on education with regard to classroom discourse, learning and instruction, for example, [27,35,36,37,38,39,40,41,42]. Furthermore, the field of Computer-Supported Collaborative Learning (CSCL) has been influenced by EMCA, since “it implies that we can observe and report on the ability of given technologies and pedagogies to mediate collaborative interaction” [43] (p. 2).
Ethnomethodology (EM) focuses on “the methods by which observable actions are produced” and seeks “to investigate how social activities are accomplished” by the participants [44] (p. 20). Conversation analysis (CA) draws from EM “a concern for understanding how order” is “achieved in social interaction”; it has developed a robust and sophisticated “empirically based methodology based on micro-analytic studies” [45] (p. 5). Particularly, CA offers a powerful tool to examine multimodal interaction as “a site where intersubjective understanding” among the participants is “created and maintained,” the analytic focus being “exclusively on meanings and understandings that are made public through conversational action” [8] (p. 156).
According to CA, communication is sequentially organized. Sequences are ordered series of turns through which participants accomplish and coordinate an interactional activity. The relevance of any turn is to be understood from its occurrence in a series of turns. The latter are unfolding in time, that is they refer to what has been said (done) before and raises expectations about relevant next turns. The most common type of sequences are dyadic adjacency pairs uttered by two different speakers producing one turn each. For example, a question (first pair part) creates a conditional relevance for an answer (second pair part), likewise a summon (calling for the attention of the other participant) “requires” a next turn in which the addressed interaction partner indicates that he/she has heard and is able to respond [33] (p. 15). It is the “understanding of the preceding turn displayed by the current speaker” which constitutes “the basis for any other type of intersubjective understanding” [8] (p. 156).
Basically, there are two possibilities in CA to construct an account of the results coming out of the investigation of a noticed phenomenon: either producing a single case analysis or establishing collections of recurring patterns of action [45]. The former involves “looking at a single conversation, or section of one, in order to track in detail the various conversational strategies and devices which inform and drive its production” [46] (p. 114). The latter is a possible next step and is used “to test the robustness of a particular description of action and to refine the analysis in the light of repeated instances of an action in different instances of interaction” [45] (p. 11) (However, “while there may be patterns which span contexts and participants, each context is unique” and so “a collection is a collection of single instances rather than multiple examples of the same thing,” which also makes (statistical) quantification highly problematic [45] (p. 11).).
The work we present in this article is best described as belonging to the first category. Indeed, we look at a single case of how three participants in a TUI-mediated JPS-activity make relevant their understandings of the functions of a salient widget (the red button) to highlight how they accomplish the related interactional work of discovering and to explore how their insights are consequential on the organization of their discovery work with regard to the science curricular aspect. As mentioned above, the phenomenon of discovery work in joint activities has already been covered by other research [4,10,11,29,30] (see Section 3 and Section 4.3.). Thus, one could also consider our here presented work as a contribution to a collection of instances of discovery work and so increases and diversifies the sites it was investigated.

4.3. Organizing the Research Process

As we already pointed out, the study was conducted in the context of an emerging cross-institutional collaboration involving researchers from both computer and social sciences. So, the interest of the video data construction was three-fold: First, exploring the strengths and the weaknesses of the designed TUI-mediated problem-solving activity; second, exploring participants meaning-making practices per se; and, third, exploring how the previous two aspects are intertwined.
After creating verbal transcripts of the video data, we started out with an “unmotivated exploration of the data” [8], that is watching the tapes repeatedly along with examining the transcripts to find out what is happening. This process sometimes involves “focusing on very small segments” and “sometimes on larger entities” [8] (p. 158). However, any observations, including those made by EMCA researchers are “informed by theories and the observer’s preconceptions” [8] (p. 158). So, basic EMCA conceptions and results from previous studies inform the exploration process and allow to “make observations that are theoretically valid and differ from what common sense can offer” [8] (p. 158).
During the process of getting familiarized with the data, something challenging arose from the data: we realized that the participants of two groups had “hacked” the red button widget to “short-circuit” the problem-solving activity. To trace this process we identified the excerpts, where the participants oriented to and made relevant their understandings of the red button. Those excerpts were then multimodally transcribed according to the conventions laid out in Appendix A. When we examined them to uncover their organizational features, we found that their sequential organization could best be described as following the one of a “discovery-in-process” [10] (p. 213). Ideally, this process starts with an initial noticing; followed by a proposal for a possible discovery; leading to an assessment where the candidate is put to the test and then either affirmed or rejected; finally, in case of a positive assessment the proposal is accepted for all present purposes of the ongoing procedure (uptake) [11]. In line with a multimodal approach, Koschman and Zemel point out that “discovery and discovering work are not exclusively linguistic phenomena” [11] (p. 32), an aspect we shall highlight in our analysis.

5. Analysis

Before analyzing the selected excerpts, we provide a brief overview on how the three participating groups coped with the problem-solving activity, and, from that, we explain why we chose the first group and how we selected the excerpts in question.

5.1. General Observations

The first group (3 participants) took in total 23:43 min to solve all the tasks correctly. Also, the second group (2 participants) managed to correctly solve the tasks but only took 17:05 min. The last group (2 participants) was the quickest one (10:27 min). However, they did not solve any of the tasks correctly. Based on the collected video data, we could observe some similarities among the groups’ approaches but more particularly between the first and the second group. During the “start” phase, all groups had a look at the different widgets and the scenes and coronas on the table. However, despite our expectations, they did not truly make use of that phase to explore the tool and realize all the functions of the widgets: group 1 and 2 quickly understood how to control parameters by rotating the widgets but did not figure out the use of the red button; and group 3 figured out the partial use of the red button but did not understand how to control the widgets (i.e., that they need to be rotated). So, for all the groups, many features were still hidden when they began to work on the first task.
Group 1 managed to figure out how to display results with the red button during task 1 (after 5:38 min) and how to view the green feedback at the end of task 2 (after 7:19 min). With these tools at hand, they then solved the tasks in a rather systematic way, verbalizing steps, forming tentative hypotheses, trying them out, drawing conclusions. After 15:00 min, they figured out that they could hold the button down and speeded up the task resolution by adjusting the widgets and relying on a mere trial and error approach. Group 2 needed 6:25 min to figure out the use of the red button, but then right away understood the three features: seeing the results, receiving feedback on the correctness and the possibility to hold it down. They then decided to go back to the first task and were able to accomplish all the tasks very quickly by holding the button down most of the time and adjusting the parameters progressively. Group 3 had most difficulties to solve the tasks. It seemed that they did not understand what exactly they were supposed to do and how they could use the table to solve the tasks. For each task, they changed some settings without engaging any further with the system reactions and quickly moved on to the next task.
Since the participants of the first group successively uncover the different functions of the red button over a series of events, the analysis of their related joint interactional work offers the richest account of discovering work with regard to one topic (the red button). Furthermore, as the detailed analysis will show, the outcome of the latter is highly relevant for the further course of their actions with regard to the science curricular discovery work targeted by the pedagogical design of the problem-solving activity. In the following, we shall provide a detailed analysis of all the instances where the red button is topicalized by the participants (excerpts 1, 2, 3, 5, 6) and of one exemplary instance of discovery work oriented to an aspect of the bicycle mechanics (excerpt 4). Furthermore, we recount how the participants organize their problem-solving activity after a relevant discovery or identification of a new function of the red button is accomplished.

5.2. The Red Button as a Potential Starting Device

After entering the room, the participants start to visually explore the semiotic space of the TUI. At first, they do this more or less independently from one another and the TUI serves as a somewhat scattered focal point of attention. A first attempt to establish a more localized shared focus is undertaken by Trish when she comments on and successively points to the coronas of the output and gear widgets and the graphical representation of the bike. Both Matthew and Jessica follow her pointing gestures with their gazes. Then, Matthew introduces a specific focus by pointing to the red button object and describing it as looking “like a button” (Figure 4, line 1–2).
Matthew is the first one who introduces the topic of the red button by pointing to the corresponding object and pointing out its similarity to a button. Upon receiving no verbal feedback from Jessica and only a short “mhm” from Trish (line 3), he elaborates on the uttered resemblance by assuming that the red button is a starting device to get to the first task (lines 6–9). His proposal is neither accepted by Jessica or Trish, nor is it openly and directly rejected. Jessica expresses her scepticism through a lateral head movement (line 11, fg. 1.2) and Trish mitigates her discarding response through a “yes but … construction” and through establishing some kind of resemblance between her and Matthew’s suggestion (“that is also like this,” line 13). More precisely, Trish points to the task wheel widget and refers to its inscription (“task one to six test A B,” line 15). In a sense, she uses the design of this particular tangible object as a clue that the red button may not be a starting device. Note that, in his concluding reply (“this means that,” line 16), Matthew only takes up the information regarding the inscription (“there are six tasks,” line 16). Both the topic of the red button and the idea of it being a starting device—explicitly expressed by Matthew with regard to the red button and vaguely expressed by Trish with regard to the task wheel—remain unresolved and so pending until its resolution in excerpt 2.
In the minute following excerpt 1 (not transcribed here), they address the remaining widgets one by one and attempt to figure out their purpose, however without trying them out. Then, Matthew multimodally restates his previous hypothesis with regard to the red button (excerpt 2, lines 1–2) and explicitly addresses his co-participants through his gaze and his smile.
This time, Trish picks up Matthew’s idea (Figure 5, “shall we?” line 4). She ratifies his assumption by projecting it as an appropriate next action. Through her formulation (“we?”) she lifts her invitation to take the action (pushing the button) up to the group level. Although Trish addresses Mathew with her gaze and smile, it remains unclear who shall actually press the red button (see the long pause of 1.5 s at the end of line 4 filled with smiles and chuckles). Matthew then announces that he will “do it” after providing an account for his initiative (“if no one wants,” line 7). Through their mutual gazes and their chuckling all the participants collectively display their support but also their uncertainty and curiosity with regard to the reaction of the system. So, Matthew presses the button (line 9) and, subsequently, briefly displays that he interprets the ensuing reaction of the TUI (line 10) as relevant by making an index up gesture (line 11, fg. 2.4). He, however, rapidly withdraws his hand and verbalizes that his hypothesis can no longer be sustained (line 12), which is smilingly echoed by Trish (line 13).
With regard to discovering work, we are witnessing here an initial noticing that also includes a multimodally embodied candidate proposal for a potential purpose of the red button (i.e., a starting device): Matthew is pointing and gazing at the physically present red button (fg. 1.1) while verbally suggesting a practical use that is potentially relevant for their actual conduct (pressing to start). However, his proposal is met with embodied scepticism (fg. 1.2), respectively with a mitigated negative verbal assessment that mobilizes the task wheel object with its symbolic inscriptions.
Since the starting device issue is pending, Matthew renews his previous candidate proposal in a similar multimodally embodied way for a function of the red button at the beginning of excerpt 2. This time, his initiative leads to a testing of his conjecture and a subsequent negative assessment. Notice the embodied nature of Matthew’s assessment: First, he briefly anticipates an affirmation of the tested proposal through his index up gesture. Then, he withdraws his hand and so withdraws his appraisal. Finally, he utters the negative outcome of the assessment procedure.
In a sense, we have a negative uptake here. The idea of the red button as a starting device is refuted and definitively off the table. They have come to know that the red button widget is not a starting device. Yet, they have introduced now the function of the red button as a recurrent topic. Since this issue is not resolved and remains pending, we shall witness in the subsequent excerpts, how they interactionally elaborate an understanding of the function of the red button through discovery work (excerpt 3, 5, 6). Furthermore, we point out how their related findings are consequential on the organization of their problem-solving activity as an actual engagement in discovery work, or as a simple working off of the assignments (excerpt 4 and narrative accounts in Section 5.4. and Section 5.5.).

5.3. The Red Button as a Result-Displaying Device

After reading the instructions of task 1 (“In which situation the biker needs to use the lowest force on pedal?”), participants start sharing their ideas about what impacts the force on pedal, manipulate the widgets and contemplate the changing coronas. The white boxes of the energy and force on pedal widgets catch their attention and they start wondering how they can trigger the display of the value. After having tried several types of manipulations (rotating, moving, tipping with the fingers on objects), Matthew refers to the red button and makes a suggestion in line 1 (Figure 6).
Matthew appears to experience a sudden insight and shares it with his co-participants. His turn is prefaced by a change-of-state token (ah) indicating that he “has undergone some kind of change in” his “locally current state of knowledge” [47] (p. 299). He multimodally produces his proposition by formulating it as a reversed polarity question [48] (p. 2) and by pointing to the relevant spots (corona of force widget and red button) on the TUI-interface. Both Trish and Jessica verbally display their agreement, which corresponds to the preferred answer to this type of questions [48]. The former through an overlapping (line 3) and the latter through an immediate affirmative response (line 4). Matthew then recycles the “display part” of his previous turn in talk (line 5) and the corresponding gesture (line 6). Trish moves her hand over the red button (line 7) when Matthew points to the corona of the force widget (line 6). So, in a sense, both participants jointly enact the two reference points of Matthews previous turn (red button and corona of force widget). By instructing Trish to press the red button (line 8), Matthew interprets her preceding hand movement and the halting position of her hand (line 7) as readiness to perform the relevant action and to try out the ratified hypothesis. Trish then promptly pushes the red button and releases it again, which triggers the brief display of the energy and force values in the corresponding coronas (line 10). Thereupon, Matthew points out that his initial proposition is now confirmed: pressing the red button does lead to the display of values in the coronas of the output widgets. To do so, he relies on the change-of-state token “ah” and the closing particle “voilà” [49] (line 11), where the latter co-occurs with a pointing gesture (line 12).
Thus, in principle, the sequence, initiated in line 1, is now completed. Yet, it is extended through Jessica’s noticing that the values are no longer displayed (line 13). Simultaneously, Trish moves her hand again to the red button and presses it after Jessica has finished her turn. However, this time, she does not release the button immediately but holds it for some time. Consequently, the values remain on display, prompting Matthew to read aloud the value of the force on pedal, so re-affirming the red button as an output displaying tool.
In this excerpt, the three participants jointly achieve a discovery through her interactional work. Matthew produces a new proposal for a function of the red button, that is, a trigger for the display of output values, at least, in the gesturally referred to corona of the force-on-pedal widget. Through his swiping and pointing movements he mobilizes the two objects (force-on-pedal widget and red button) and visually establishes a relationship among them that he verbalizes as conditional. Matthew’s candidate proposal is then put to the test through enactment and assessed positively (lines 8–12). Furthermore, the trial is repeated to reproduce its effect (13–18) after one of them notices the disappearance of the triggered display. They have now discovered a first function of the red button device and, in the following excerpt, we can witness how they take it up to work on the accomplishment of the given scientific problem-solving task.
Following Matthews instruction (lines 1), Trish pushes the red button (line 3). Matthew’s co-occurring pointing gesture to the corona of the force widget (line 2) creates a local focus of attention that all participants chose to share (see fg. 4.1). When the TUI displays the output values (force and energy, line 4), Matthew interprets them as not meeting the demand by suggesting turning each gear to its other extreme (line 6). They then jointly carry out this adjustment: Trish acknowledges Matthew’s suggestion by releasing the red button and instructs him to turn the object of the pedal gear widget (line 9), he is holding already (line 7). Matthew follows her instruction and sets the pedal gear to the smallest one (line 10). While Trish moves her hand again to the red button and so gets ready to check the result (line 13), Jessica seizes the wheel gear (line 12) and seeks approval to set it to a bigger gear (line 11). Matthew ratifies and specifies that she should set the wheel gear to its biggest extension. Right after his affirmation, she starts shifting the wheel gear (line 15). During Jessica’s manipulation, Trish retracts her hand (line 16) and only moves it back after Jessica has finished her adjustment (line 17). Then, she pushes the button (line 17) and the TUI displays the output values. The latter are significantly lower which is noted and commented by all the participants. Trish points out that the values are now “even less” than previously (line 19), which is acknowledged in line 20 by Matthew (“yes”), who further indexically (“this way”) verbalizes their conclusion (“then”) contrasting it (“nonetheless”) to a previous setting (at the beginning of the excerpt the pedal gear is set to its biggest and the wheel gear to its smallest extension). Simultaneously, Jessica underlines the epistemic status of their finding, stating that they “know it” (line 21) and Trish releases the red button (line 23). Matthew acknowledges again and reformulates their finding by pointing out that the current gear setting corresponds to the one requiring the lowest force (line 23). To make sure of this again, he briefly pushes the red button. After Matthew utters “yes,” Jessica also produces an acknowledgement and reaches for the task wheel to proceed to the next task. Yet, she retracts her hand again when Mathew continues reformulating their finding (“so it’s from small to big,” line 27) while pointing to the wheel gear. Then, Jessica again attempts to conclude by rhetorically asking to move on (line 29) and does so without waiting for someone else’s approval (line 31). Notice that although Matthew provides the preferred answer (“yes”), he prefaces it with “eh” and produces it with some delay. Considering this markers and Matthew’s pointing movement to the corona of the energy widget, we may assume that he might have preferred lingering on the question.
Extract 4 (Figure 7) is interesting in two ways. First, it demonstrates how their acceptance of the proposal that the red button is (only) a result-displaying device is relevant for them for the present purpose and becomes an instrument of dealing competently with their momentary understanding of the task. Second, it shows how—through discovering work—they deal with and come to grips with an often-misconceived concept, that is, that the force on pedal decreases when selecting a larger sized gear at the wheel but increases with a larger sized gear at the pedal. To do so, they witnessably produce multimodally embodied discovery work. Mathew introduces a candidate proposal for the solution of the task (“In which situation the biker needs to use the lowest force on pedal?”), namely to inverse the adjustments of the gears (line 6). The latter is not only talked into being but also enacted through a spinning hand movement (fg. 4.3). Again, they jointly set the parameters accordingly and get ready for the ensuing assessment. After their trial, they jointly interpret the outcome of their trial—a significant reduction of the force on pedal—as a successful accomplishment of the task and so affirm their candidate.
However, the reader will have noticed that, so far, the participants have not yet solved the task as expected. To do this, they still have to reduce the coarseness of the ground from “sand” to ‘”treet.” So far, they have not realized their "mistake" since they have not figured out yet that the red button also serves to visualize whether the independent variables are set correctly or not with respect to the task. That is what we are going to see in the following excerpt.

5.4. The Red Button as a Checking Tool

After the previous extract, the group works on the second task (“In which situation the biker consumes the most energy?”). They invert the gears and progressively change the inclination and the distance to their respective maxima and the ground to the surface they esteem to be “more exhaustive” (i.e., sand). After all these adjustments, Trish presses the red button and this time a green belt around the button and a green tick flash (Figure 8, fg. 5.1).
This extract shows us how they discover that there is more to the red button, namely that it is the green marks (tick and green belt) that signal the successful accomplishment of the task. Matthew notices the green tick (lines 2–3) and marks it as a new information (“oh”) and Trish points to the green belt (line 4). In lines 6 and 7, Mathew ratifies (“oh cool”) Trish’s contribution (the green belt) and signals through the use of “ok” that they have completed something and, consequently, Trish suggests moving on (line 9) to the next project which is working on the next task. However, Trish displays that she has gained an insight (line 8). She interrupts Jessica’s proposal to start working on the next assignment by sharing her conclusion that their solution to the previous task was wrong (see extract 4). Matthew responds to this announcement with wonder (line 11). So, Trish reaffirms her insight by providing a multimodally embodied explanation: she draws a tick in the air with her right index (fg. 5.2). Towards the end of her turn, she addresses Jessica with her gaze. This affiliation seeking conduct "pays off" when they jointly respond (lines 15–16) to Matthew’s second wondering (line 14) and simultaneously verbalize their argument (lines 15–16) why the previous task was not solved properly.
They then (passage not transcribed here) return to the first task. After restoring the settings, they had set at the end of their presumed resolution of the first task, they quickly recognize that they also have to adjust the ground to the “hard ground” (i.e., the asphalt). After checking their now correct solution by pushing the red button, they collectively express their pleasure on sight of the flashing green tic (and belt) and move on to the next assignment.
Again, we can see, how they organize their activity as discovering work. We have an initial noticing (their solution to the previous task was wrong) through Trish, who then embodies a candidate proposal for a new function of the red button (a green tick signals that the solution is correct). The latter is jointly affirmed by Trish and Jessica through their overlapping talk (line 15 & 16) and by Matthew through his acceptance of returning to the previous task. Since the proposal is immediately followed here by an affirmation, this episode may be regarded as a weaker example of a discovery sequence, perhaps better described as an instance of a recognition [11] (p. 43). Nevertheless, the positive assessment is followed by a highly consequential uptake in their situated activity. They return to the previous task and resolve it by relying on their newly gained knowledge about the red button. Their corresponding competence can also be witnessed in the subsequent narrative account of their resolution of the third task (“How can the biker consume the same energy as in task 2 but minimizing the required force?”).
Indeed, they rely in a systematic way on their so-far gained knowledge concerning the result-displaying and checking features of the red button. While doing so they engage in intensive discovery work. They continuously take the red button widget up as tool to move on with and to finally solve the assigned task. They name the problem, form tentative hypotheses, try out the latter, draw conclusions and, if necessary, restart the process. More precisely, they take up proposals from previous task resolutions (e.g., 08:24:00 TRI: the force on pedal must go down (.) how did we turn that [pedal gear] [...] we had it very big); they utter initial noticings (e.g., 08:59:40 MAT: this means that we have to reduce anything) and proposals for possible solutions (e.g., 09:05:40 MAT: either [we reduce] the inclination and we switch to another [wheel] gear there; 09:05:40 TRI: or here one gear up); they assess and put to the test their hypotheses (e.g., 10:27:32 MAT: [...] we have to [...] experiment somehow if we don’t know it physically); and, finally, they figure out the required settings, leading again to a joint cheering on sight of the green corona of the red button, which marks their adherence to the co-achieved resolution and their, for their present purposes, co-constructed reality.

5.5. The Red Button as a Fast-Tracking Tool

When they work on the fourth task they continue relying on the red button to inquire about the outcome of their trials. However, they considerably increase the pace: Solving the previous third task took them 3 min and 48 s, during which they pressed the red button 21 times to check the results. To solve the fourth task, they press the red button for twenty times during a period of 2 min and 17 s. They consider manipulating a widget and pressing the red button in a chronological order: first they try with one or more widgets and then they verify with the red button. However, this pattern changes in the 15th minute, when they, more or less accidently, find out how the red button can be used to have instant access to the results while manipulating the input widgets.
When Jessica has finished adjusting the inclination of the road, they all look at the centre of the TUI where the red button is located. Through her so-prefacing Jessica then verbally initiates the next action (Figure 9, line 1), namely pressing the red button to check if their candidate solution for the assigned task is correct. The latter is done by Matthew (line 2); notice that he will now keep holding down the button until the end of this sub-excerpt. The TUI “responds” by displaying both the output values and the grey belt around the red button (line 3), indicating that the task is not solved yet. Jessica notices that the energy is below the target. She proposes to increase the distance again (line 4) (they had decreased it in previous attempts) and, at the same time, sets it to its maximum (line 5). Matthew simultaneously makes the same suggestions, so both align with one another. After Jessica has set the distance to its maximum (30 km), Matthew does something peculiar: he pushes the red button, he is already holding down. With regard to their antecedent procedure, we can say that he is following the “set and check” pattern they have previously established upon discovering the checking function of the red button widget. The circumstance that Mathew, after briefly releasing and re-pushing the red button, continues holding it down, will now “trigger” the detection of a new function of the red button.
When Jessica follows Matthew’s instruction to try out what happens when she turns the distance object (while he is still holding the red button), he multimodally displays his sudden insight that it is indeed possible to manipulate the widgets while holding down the red button (Figure 10, lines 9–13). He moves his right hand towards his face (without touching it) while producing a deprecating utterance (“how stupid”) and then then points to the red button while verbalizing his finding (“you can hold it”). Jessica “chimes in” to complete the second part of the utterance with him (“can hold it”), marking her shared understanding of his finding. Matthew’s deprecating pre-face is in-part addressed to himself and in-part to the whole group. He does point to himself but does not complete the gesture (i.e., touching his forehead) and he omits naming himself as the recipient (e.g., “of me”). Furthermore, neither of his co-participants is producing a disagreement, which would be the preferred response to a self-deprecation.
In a sense, Mathew’s turn in line 13 comprises a noticing (“how stupid”) and a proposal that is simultaneously an assessment (“yes ok you can hold it”), that is immediately affirmed by his co-participants. Hence, this instance of discovering work is also better described as a recognition or an identification of the last feature of the red button. Here, the participants accountably downgrade their “discovery” to a long-overdue identification of an easily detectable embedded feature of the red button and display their embarrassment about their “poor” performance (i.e., not having figured out this obvious function much earlier in the process). They then immediately produce an uptake by integrating the new function in their proceeding task accomplishment and quickly find a correct solution (lines 16–26). Their collective cheering (line 27) and Jessica’s recycling of “stupid” (while reaching for the task wheel) closes this episode, as well as the discovering work regarding the functions of the red button and, soon thereafter, also regarding the curricular problem-solving activity.
Indeed, from now on they will rely on this newly discovered feature to considerably speed up the problem-solving process during the last two tasks. When Matthew suggest holding down the red button, they start increasing the pace of manipulating the widgets. At the beginning they still formulate some insights they recall or experience while watching the changing energy and force values (e.g., “no look the kilometres just change the energy,” “nothing changes with the force on pedal”). Nevertheless, their talk becomes increasingly indexical (e.g., “that one smaller,” “this one down, that one up”) and appears only to be oriented toward a fast task completion. At the end, they are even surprised when the green belt and the tick flash and Matthew utters “how did we do that? nobody knows, it doesn’t matter, okay good”). Thus, the participants’ conduct is mainly oriented to a fast task completion, pushing discovering work about mechanisms to the background.

5.6. The Process of the Second Group

As was mentioned before, particularly the second group showed similarities in their procedure of discovering the functions of the red button and accomplishing the tasks. For this reason, we provide a brief account here of how they proceeded. Already after 30 s, one participant of the second group mobilizes the red button but he does only so by touching and pressing it without verbally topicalizing it. The latter occurs twice: two respectively three minutes later, when the same participant raises the question about the purpose of the red button and the second participant displays ignorance (“I don’t know” respectively head shaking). In the following, the first participant puts his hand again over the red button but does not touch it. At this stage, the first group had uttered a hypothesis (the red button is a starting device) but common to both is that they not discover a (viable) feature of the widget in question. This happens approximately two minutes later, when both participants make their first understandings of the functions of the red button accountable. The first participant notices that the output values are displayed and the second infers that they have solved the previous assignments incorrectly (they are now at task 3). Consequently, they return to the first task and re-solve it within a minute. So, similarly to the first group, they establish the red button as a result-playing device. Upon returning to the second assignment they notice the green corona for the first time and infer from this that their current task is solved and the previous one was not. So, they go back to the first assignment and stay with it for one minute. During this time, the first participant makes verbally accountable this new insight (a green corona means that the answer is correct and a grey corona means that the answer is incorrect) and points to it as a facilitating strategy. Both is ratified by the second participant. When they return to the third assignment the first participant utters his insight “Basically now we just can fiddle until it works.” That is what they will do for most of the remaining time, leading to a considerable increase in the pace of the achievement of the remaining tasks, as it was the case for the first group.
In this article, we focused on a single case analysis and, consequently, produced an account of the multimodal organization the discovering process of the first group alone. Yet, the above summarization indicates that our work can be the first step into the establishment of a collection with regard to the accomplishment and the organization of discovery work in learning settings mediated through horizontal shared interfaces.

6. Discussion and Conclusions

Through our micro-analysis of the multimodally embodied conduct of three students engaged in a TUI-mediated joint problem-solving activity, we reconstructed how they interactionally accomplish doing discoveries in situ. While constantly displaying a joint orientation to their task, they are doing a multimodally embodied work of noticing, of directing the other’s attention, of seeking and securing mutual understanding [11]. Finally, new insights are introduced in their shared world. In all instances, we can find the three-part sequence of proposal (sometimes preceded by a noticing), assessment and uptake as pointed out by Koschmann and Zemel [11]. They further specify that it is “what comes between the proposal and the uptake that makes the participants’ conduct recognizably a discovery” [11] (p. 43). This applies to excerpts 2, 3 and 4, where the proposed hypotheses are put to the test through trials which are then assessed, whereas excerpts 5 and 6 may be better described as recognitions or identifications since the proposals are immediately affirmed. Nevertheless, there is “no discontinuity between the two” but rather “a continuum of action organizations ranging from the simplest forms of identification on up to the most sophisticated forms of scientific evaluation” [11] (p. 43).
Furthermore, through our analysis we highlighted that the participants’ discovering work were oriented to two different foci: first, the functions of the widgets and here particularly the red button, and, second, the resolution of the different assignments mediated through the TUI-activity. The participants have transformed an initially unknown object (the red button) into an effective instrument to put their hypotheses to the test. Indeed, prior to identifying that the red button can be held down to get instant access to the results while manipulating the input variables, they rely on the tangible objects, scenes and coronas to engage in discovering work with regard to the underlying bicycle mechanics model. They name problems, form tentative hypotheses, put them to the test, draw conclusions and, if necessary, restart the process. However, after recognizing what we called the “fast-tracking” function, we witness a conduct that a participant of the second group had termed “fiddling,” i.e., a quick trial and error approach exhibiting predominantly indexical and minimalistic linguistic forms. This “fiddling” conduct prevents them from engaging with the underlying physics or bicycle mechanics aspects. From a pedagogical point of view, this does certainly not correspond to a valuable outcome. Paradoxically, however, the fast-tracking procedure also corresponds to a perfecting of the mobilization of the red button widget to maximize task performance in terms of speed. In other words, they outwitted the underlying system mechanics to efficiently do what they were asked to do, namely solving the tasks.
Nevertheless, as we were able to show in our analysis, some design aspects embedded into the red button widget were, at least, partly working since they contributed to the organization of the activity as more scientific inquiry work. One such a design principle was the decoupling of the input widgets from the calculation of the results. Indeed, Hornecker [50] argues that complex domains are forced to violate Ishii’s principle of direct and continuous mappings [51], in order to facilitate reflection and collaborative sense-making. Our analysis is in line with this claim, since it is only when the participants restored the coupling that their discovering work ceased. Another design aspect is that the mechanical design requires active physical pressure to activate it. Other researchers used comparable spatial or physical properties (e.g., spatially separated stations [52] or whole-body interaction on large physical spaces [53]) to increase the burden for participants to use certain features in a tangible system, in particular those features which allow for restoring a coupling and through this, proceed with non-reflected trial and error conduct. So, in order to inhibit the use of the red button as a fast-tracking tool, one would have to render the decoupling more difficult or “costly” for the participants, for example through limiting the periods or the number of times the button can be pushed, through increasing the physical resistance of the button mechanism, or through partially disabling the input widgets while the red button is pressed.
Finally, our study thus contributes to a better understanding of how discoveries are interactionally accomplished within the context of ICT mediated joint problem-solving/discovery-learning activities, of how designed objects (such as the red button) are appropriated (partly through discovery work) by the participants to cope with the tasks at hand and of how both aspects are intertwined. So, our findings can inform developers to provide affordances to encourage learners to engage with and discover the underlying mechanisms of a simulation and burden non-reflected trial and error strategies.

Author Contributions

Conceptualization, P.S., B.A. and V.M.; Methodology, B.A.; Software, V.M.; Writing–original draft, P.S., B.A. and V.M.; Writing–review & editing, P.S.


The original bike Microworld was designed and developed in the scope of the ERASMUS+ Reengage project. The user study reported in this paper received no external funding.


We would like to thank Jessica, Matthew, Trish and the other study participants; Josef Camilleri, Christopher Cassar, Florian Islamaj, Oiane Martínez Lόpez and Irene Soria Lόpez for the initiation of the bicycle simulation idea within the context of a ReEngage training event organized in January 2016 in Madrid; and Timothy Koschmann for his valuable comments.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Note on transcription conventions

JES/MAT/TRI: participant speaking or doing the embodied action
TUI display of tangible user interface
RB red button widget
((pushing the button)) non-verbal conduct
? question
[ overlapping talk
(0.6) timed pause
(.) very short pause
fg. frame grab


  1. De Raffaele, C.; Smith, S.; Gemikonakli, O.; Nygaard, C. The Application of Tangible User Interfaces for Teaching and Learning in Higher Education. In Innovative Teaching and Learning in Higher Education; Hayes, S., Hørsted, A., Eds.; Libri Publishing: Faringdon, UK, 2017; pp. 215–226. ISBN 9781911450085. [Google Scholar]
  2. Plass, J.L.; Schwartz, R.N. Multimedia Learning with Simulations and Microworlds. In The Cambridge Handbook of Multimedia Learning; Mayer, R.E., Ed.; Cambridge University Press: Cambridge, UK, 2014; pp. 729–761. ISBN 9781107610316. [Google Scholar]
  3. De Jong, T. Computer Simulations. Technology Advances in Inquiry Learning. Science 2006, 312, 532–533. [Google Scholar] [CrossRef] [PubMed]
  4. Sunnen, P.; Arend, B.; Maquil, V. “Okay, yes it’s true”—Doing discovering work in a tangible-user-interface-mediated joint problem solving physics activity. In EDULEARN17, Proceedings of the 9th International Conference on Education and New Technologies, Barcelona, Spain, 3–5 July 2017; IATED: Valencia, Spain, 2017; pp. 8057–8068. [Google Scholar]
  5. RE-ENGAGE Research Project Website. Available online: (accessed on 12 September 2018).
  6. Sacks, H.; Schegloff, E.A.; Jefferson, G. A simplest systematics for the organization of turn taking in conversation. Language 1974, 5, 596–735. [Google Scholar]
  7. Garfinkel, H. Studies in Ethnomethodology; Prentice Hall: Englewood Cliffs, NJ, USA, 1967. [Google Scholar]
  8. Peräkylä, A. Conversation Analysis. In Qualitative Research Practice; Seale, C., Gobo, G., Gubrium, J.F., Silverman, D., Eds.; Sage: London, UK, 2006; pp. 153–167. [Google Scholar]
  9. Mondada, L. The Conversation Analytic Approach to Data Collection. In The Handbook of Conversation Analysis, Sidnell, J., Stivers, T., Eds.; Wiley-Blackwell: Oxford, UK, 2013; pp. 32–56. ISBN 9781118941294. [Google Scholar]
  10. Koschmann, T.; Zemel, A. Optical pulsars and black arrows: Discoveries as occasioned productions. J. Learn. Sci. 2009, 18, 200–246. [Google Scholar] [CrossRef]
  11. Koschmann, T.; Zemel, A. ‘So that’s the ureter.’ The Informal Logic of Discovering Work. Ethnogr. Stud. 2011, 12, 31–46. [Google Scholar]
  12. Pontual Falcão, T.; Price, S. Interfering and resolving: How tabletop interaction facilitates co-construction of argumentative knowledge. Int. J. Comput.-Support. Collab. Learn. 2011, 6, 539–559. [Google Scholar] [CrossRef]
  13. Rogers, Y.; Sharp, H.; Preece, J. Interaction Design. Beyond Human-Computer Interaction, 3rd ed.; Wiley: Chichester, UK, 2011; ISBN 9780470665763. [Google Scholar]
  14. Zaman, B.; Vanden Abeele, V.; Markopoulos, P.; Marshall, P. Editorial: The evolving field of tangible interaction for children: The challenge of empirical validation. Pers. Ubiquitous Comput. 2012, 16, 367–378. [Google Scholar] [CrossRef]
  15. Fernaeus, Y.; Tholander, J.; Jonsson, M. Towards a new set of ideals: Consequences of the practice turn in tangible interaction. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, New York, NY, USA, 18–20 February 2008; ACM: New York, NY, USA, 2008; pp. 223–230. [Google Scholar]
  16. Scott, S.D.; Sheelagh, M.; Carpendale, T.; Inkpen, K.M. Territoriality in collaborative tabletop workspaces. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, Chicago, IL, USA, 6–10 November 2004; ACM: New York, NY, USA, 2004; pp. 294–303. [Google Scholar]
  17. Rogers, Y.; Lim, Y.; Hazlewood, W.R.; Marshall, P. Equal opportunities: Do shareable interfaces promote more group participation than single users displays? Hum.-Comput. Interact. 2009, 24, 79–116. [Google Scholar] [CrossRef]
  18. Edwards, L.D. The Design and Analysis of a Mathematical Microworld. J. Educ. Comput. Res. 1995, 12, 77–94. [Google Scholar] [CrossRef]
  19. Maquil, V.; Tobias, E.; Anastasiou, D.; Mayer, H.; Latour, T. COPSE: Rapidly Instantiating Problem Solving Activities based on Tangible Tabletop Interfaces. Proc. ACM Hum.-Comput. Interact. 2017, 1, 16. [Google Scholar] [CrossRef]
  20. Maquil, V.; Latour, T. TULIP: A widget-based software framework for tangible tabletop interfaces. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Duisburg, Germany, 23–26 June 2015; ACM: New York, NY, USA, 2015; pp. 216–221. [Google Scholar]
  21. Jonassen, D.H. Supporting Problem Solving in PBL. Interdiscipl. J. Probl.-Based Learn. 2011, 5, 95–112. [Google Scholar] [CrossRef][Green Version]
  22. De Jong, T.; Van Joolingen, W.R. Scientific discovery learning with computer simulations of conceptual domains. Rev. Educ. Res. 1998, 68, 179–202. [Google Scholar] [CrossRef]
  23. Atkinson, P.; Delamont, S. Mock-ups and Cock-ups: The Stage-Management of Guided Discovery Instruction. In School Experience. Explorations in the Sociology of Education; Woods, P., Hammersley, M., Eds.; Routledge: London, UK; New York, NY, USA, 2017. [Google Scholar]
  24. Dewey, J. The Theory of Inquiry; Holt: New York, NY, USA, 1938. [Google Scholar]
  25. Bruner, J. The Act of Discovery. Harv. Educ. Rev. 1961, 31, 21–32. [Google Scholar]
  26. Neber, H. Discovery Learning. In Encyclopedia of the Sciences of Learning; Seel, N., Ed.; Springer: Berlin, Germany, 2012; pp. 1009–1012. ISBN 9781441955036. [Google Scholar]
  27. Seedhouse, P. The Interactional Architecture of the Language Classroom: A Conversation Analysis Perspective; Blackwell: Oxford, UK, 2004; ISBN 9781405120098. [Google Scholar]
  28. Leonard, D.C. Learning Theories: A to Z; Greenwood Pub Group: Westport, CT, USA, 2002; ISBN 9781573564137. [Google Scholar]
  29. Koschmann, T. Discovery as an Interactional Matter. Available online: (accessed on 13 August 2018).
  30. Koschmann, T.; Zemel, A. Optical Pulsars and Black Arrows: Discovery’s Work in ‘Hot’ and ‘Cold’ Science. In Proceedings of the International Conference of the Learning Sciences (ICLS 2006), Bloomington, IN, USA, 27 June–1 July 2006; ISLS: Bloomington, IN, USA, 2006; pp. 356–362. [Google Scholar]
  31. Streeck, J.; Goodwin, C.; LeBaron, C. Embodied Interaction in the Material World: An Introduction. In Embodied Interaction. Language and Body in the Material World; Streeck, J., Goodwin, C., LeBaron, C., Eds.; Cambridge University Press: Cambridge, UK, 2011; pp. 1–26. ISBN 9781107630420. [Google Scholar]
  32. Arend, B.; Sunnen, P.; Fixmer, P.; Sujbert, M. Perspectives do matter—Expanding Multimodal Interaction Analysis with Joint Screen. Classr. Discourse 2014, 5, 38–50. [Google Scholar] [CrossRef]
  33. Psathas, G. Conversation Analysis. The Study of Talk-in-Interaction; Sage: London, UK, 1995; ISBN 0803957475. [Google Scholar]
  34. Garfinkel, H. Ethnomethodology’s Program. Working Out Durkheim’s Aphorism; Rowman & Littlefield: Lanham, MD, USA, 2002; ISBN 9780752516427. [Google Scholar]
  35. Gardner, R. Conversation Analysis in the Classroom. In The Handbook of Conversation Analysis; Sidnell, J., Stivers, T., Eds.; Wiley Blackwell: Oxford, UK, 2013; pp. 593–611. [Google Scholar]
  36. Waring, H.Z. Theorizing Pedagogical Interaction. Insights from Conversation Analysis; Routledge: London, UK, 2016; ISBN 9781138086104. [Google Scholar]
  37. Stevens, R. Learning as a member’s phenomenon. Natl. Soc. Stud. Educ. 2010, 109, 82–97. [Google Scholar]
  38. Koschmann, T. Conversation Analysis and Learning in Interaction. In The Encyclopedia of Applied Linguistics; Chapelle, C.A., Ed.; Blackwell: Oxford, UK, 2013; 6p. [Google Scholar] [CrossRef]
  39. Koschmann, T.; Zemel, A.; Neumeister, M. “Case n’ Point”: Discovering Learning in the Nonce. In Learning and Becoming in Practice, Proceedings of the 11th International Conference of the Learning Sciences, Boulder, CO, USA, 23–27 June 2014; Polman, J.L., Kyza, E.A., O’Neill, D.K., Tabak, T., Penuel, W.R., Jurow, A.S., O’Connor, K., Lee, T., D’Amico, L., Eds.; ISLS: Boulder, CO, USA, 2014; pp. 551–557. [Google Scholar]
  40. Melander, H. Trajectories of Learning. Embodied Interaction in Change; Acta Universitatis Upsaliensis: Uppsala, Sweden, 2009; ISBN 978–91-554-7525-3. [Google Scholar]
  41. Pallotti, G.; Wagner, J. (Eds.) L2 Learning as Social Practice: Conversation-Analytic Perspectives; University of Hawai’i: Honolulu, HI, USA, 2011; ISBN 9780980045970. [Google Scholar]
  42. Macbeth, D. Understanding understanding as an instructional matter. J. Pragmat. 2011, 43, 438–451. [Google Scholar] [CrossRef]
  43. Stahl, G. Ethnomethodologically informed. Int. J. Comput. Support. Collab. Learn. 2012, 7, 1–10. [Google Scholar] [CrossRef][Green Version]
  44. Hester, S.; Francis, D. An Invitation to Ethnomethodology. Language, Society and Interaction; Sage: London, UK, 2004; ISBN 9780761966425. [Google Scholar]
  45. Liddicoat, A.J. An Introduction to Conversation Analysis, 2nd ed.; Continuum: London, UK, 2011; ISBN 9781441189349. [Google Scholar]
  46. Hutchby, I.; Wooffitt, R. Conversation Analysis, 2nd ed.; Polity Press: Cambridge, UK, 2008; ISBN 9780745638669. [Google Scholar]
  47. Heritage, J. A change-of-state token and aspects of its sequential placement. In Structures of Social Action; Atkinson, M., Heritage, J., Eds.; Cambridge University Press: Cambridge, UK, 1984; pp. 299–345. ISBN 9780521318624. [Google Scholar]
  48. Koshik, I. Beyond Rhetorical Questions: Assertive Questions in Everyday Interaction; John Benjamins Publishing Company: Amsterdam, The Netherlands; Philadelphia, PA, USA, 2005; ISBN 9781588116321. [Google Scholar]
  49. Bruxelles, S.; Traverso, V. Usage de la particule voilà dans une réunion de travail: Analyse multimodale. In Les Marqueurs Discursifs dans Les Langues Romanes; Drecher, M., Frank-Job, B., Eds.; Peter Lang: Frankfurt am Main, Germany, 2006; pp. 71–92. ISBN 9783631534151. [Google Scholar]
  50. Hornecker, E. Beyond Affordance: Tangibles’ Hybrid Nature. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction, Kingston, ON, Canada, 19–22 February 2012; pp. 175–182. [Google Scholar]
  51. Ishii, I. Tangible bits: Beyond pixels. In Proceedings of the 2nd International Conference on Tangible and Embedded Intreaction (TEI ’08), Bonn, Germany, 18–20 February 2008. [Google Scholar]
  52. Antle, A.N.; Wise, A.F. Getting down to details: Using theories of cognition and learning to inform tangible user interface design. Interact. Comput. 2013, 25, 1–20. [Google Scholar] [CrossRef]
  53. Stanton, D.; Bayon, V.; Neale, H.; Ghali, A.; Benford, S.; Cobb, S.; Ingram, R.; O’Malley, C.; Wilson, J.; Pridmore, T. Classroom collaboration in the design of tangible interfaces for storytelling. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems, Seattle, WA, USA, 31 March–5 April 2001; pp. 482–489. [Google Scholar]
Figure 1. (a) The Bike Microworld and its Widgets; (b) Tangibles.
Figure 1. (a) The Bike Microworld and its Widgets; (b) Tangibles.
Mti 02 00076 g001
Figure 2. The Red Button widget.
Figure 2. The Red Button widget.
Mti 02 00076 g002
Figure 3. Joint Screen.
Figure 3. Joint Screen.
Mti 02 00076 g003
Figure 4. Transcription of excerpt 1 (00:57:28–01:10:10) (see Appendix A for conventions).
Figure 4. Transcription of excerpt 1 (00:57:28–01:10:10) (see Appendix A for conventions).
Mti 02 00076 g004
Figure 5. Transcription of excerpt 2 (02:02:00–02:12:44).
Figure 5. Transcription of excerpt 2 (02:02:00–02:12:44).
Mti 02 00076 g005
Figure 6. Transcription of excerpt 3 (05:34:22 – 05:53:24).
Figure 6. Transcription of excerpt 3 (05:34:22 – 05:53:24).
Mti 02 00076 g006
Figure 7. Transcription of excerpt 4 (05:57:12 – 06:24:00).
Figure 7. Transcription of excerpt 4 (05:57:12 – 06:24:00).
Mti 02 00076 g007
Figure 8. Transcription of excerpt 5 (07:19:00–07:34:00).
Figure 8. Transcription of excerpt 5 (07:19:00–07:34:00).
Mti 02 00076 g008
Figure 9. Transcription of excerpt 6a (15:09:00–15:13:14).
Figure 9. Transcription of excerpt 6a (15:09:00–15:13:14).
Mti 02 00076 g009
Figure 10. Transcription of excerpt 6b (15:13:38–15:27:44).
Figure 10. Transcription of excerpt 6b (15:13:38–15:27:44).
Mti 02 00076 g010

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (
Multimodal Technologies Interact. EISSN 2414-4088 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top