1. Introduction and Background
Young children’s everyday lives are increasingly permeated by an array of digital technologies which are rapidly changing their experiences of play and the forms in which they make meaning [1
]. A growing area is the rise of ‘virtual reality’, ‘augmented reality’ and ‘mixed reality’, where technologies merge the digital and non-digital to varying degrees, creating new opportunities and environments for play [5
]. Whilst virtual reality might represent one end of a continuum from digital to non-digital play [7
], institutions such as LEGO House can be seen as layering digital and non-digital elements in innovative ways. In such hybrid spaces, the distinctions between digital and non-digital become increasingly hard to discriminate. As a result, contemporary play often “moves fluidly across boundaries of space and time in ways that were not possible in the pre-digital era” [8
Emerging research has suggested that young children are particularly engaged by what appears to be a ‘magical link’ between digital and non-digital domains and that multimodal technologies might enhance children’s play in new ways [6
]. Whilst health and safety concerns are one ongoing topic of discussion [9
], the widespread popularity of ‘augmented reality’ apps such as Pokémon Go and the growing use of such apps by young children suggest this field is likely to continue to expand [5
]. Despite this, much current research on children’s use of ‘mixed reality’ has developed in the disciplines of computer sciences and Human-Computer Interaction, tending to focus on children over ten years of age, and has mostly explored school-based learning [10
]. This suggests that theoretical and methodological innovations are required in order to explore the complexity of play in hybrid spaces and the significance for learning [5
1.1. Multimodal Technologies and Learning
Children’s playful interaction with digital technologies has tended to prompt polarized responses, ranging from moral panics to resounding endorsements [4
]. However, broadened perspectives on ‘literacy’ and ‘literacies’ argue for the need to recognize the significance of digital technologies in children’s everyday lives and to critically consider the role of the digital in relation to learning [13
]. In addition to querying the position of digital technology within the education system [16
], there have been calls to consider the place of digital technologies in informal spaces that exist outside or between the school and the home [17
]. It is argued that ‘third space’ settings, such as museums and makerspaces, invite particular possibilities for valuing and fostering dynamic multimodal literacies in meaningful ways [18
], placing children in active, creative and critical roles. Defined as spaces in which participants create artifacts using a range of tools and resources, both digital and non-digital, makerspaces can be seen as part of a wider rise in maker culture which values creativity, innovation and ‘learning through doing’ [19
]. Within such approaches, digital technologies are valued for their particular creative possibilities, used purposefully in combinations with non-digital forms, rather than instrumental or ‘tokenistic’ uses [15
]. It has been argued that through provision of appropriate environments, tools and materials, both digital and non-digital, children might be supported to become competent multimodal designers and re-designers, equipped for the digital age [19
]. In this article, LEGO House is considered as a form of ‘third space’ makerspace combining digital and non-digital resources into its play-based design.
1.2. A Multimodal Social Semiotic Perspective
In order to examine the potential for learning in LEGO House, apt theoretical perspectives are required which consider meaning-making with both digital and non-digital resources, and which understand that learning happens in moments and places not necessarily positioned as teaching. A multimodal social semiotic perspective on learning recognises that every act of sign-making has transformative, agentive meaning-making at its very core [1
]. This perspective suggests that every time a sign is created, meaning is made anew, and that in this act of making the sign-maker’s own understanding is reshaped [22
]. In this way, sign-making (whether with digital or non-digital resources) is recognised as ‘transformative engagement’, transforming both the outward signs and inner signs of the sign-maker, and so can be considered learning [22
Furthermore, a multimodal
social semiotic perspective highlights that such signs are always made in a multiplicity of modes, which might include image, sound, moving image, gesture, layout and so on, as well as speech and writing [1
]. From such a perspective, children use multiple modes not to compensate for emergent language, but because they intend to convey the richest meanings possible with the means available [23
]. Each mode available offers particular affordances that shape meaning-making in distinct ways and with implications for how these signs of learning are made apparent. In this way, play with a range of materials (e.g., digital and non-digital) might be seen to expand meaning-making, as “the combination of modes amplifies and/or complicates the separate strands of monomodal meanings” [25
A multimodal social semiotic perspective on learning is particularly relevant since it has been suggested that children readily switch between modes to convey their messages and meanings, and are less encumbered by cultural expectations for uses of materials than adults [25
]. Kress argues that moving between media and modes is a naturally synaesthetic, creative tendency which becomes increasingly suppressed in adulthood, but needs to be rediscovered and reinstated to support different possibilities of engagement in the world, and particularly within a changing landscape of communication [23
A multimodal social semiotic perspective therefore has the potential to offer a balanced, evaluative approach to the playful learning experiences of LEGO House by supporting detailed insights into the design and use of toys and spaces, both digital and non-digital. Multimodality offers the recognition of meaning-making occurring in many modes beyond speech and writing, and focuses attention on the affordances
of such modes to identify both potentials and constraints [2
], as well as the consequence of transduction
whereby meaning is re-made across modes [21
]. From such a perspective, toys such as LEGO can be considered semiotic resources, where their design is recognised to be socially-shaped with distinct potentialities for meaning-making [29
]. Multimodality also provides a frame for considering how such toys are used, recognising the agency and interest players bring to a play experience, thereby making meaning anew through the ways toys are used in play [23
The research visit to LEGO House was part of a Short Term Scientific Mission funded by the DigiLitEY Project, an EU COST Action investigating the digital literacy and multimodal practices of young children. The visit was hosted by the University of Southern Denmark and addressed the following combined research aims:
(1) To develop multimodal social semiotic theoretical perspectives on children’s digital and non-digital play in LEGO House
(2) To consider the methodological possibilities of collaborative research-design partnerships regarding digital and non-digital play
(3) To examine the practical ways in which play with digital technologies is designed and built into the LEGO House play-space
This article shares insights from the visit to LEGO House in order to identify the affordances of multimodal technologies, both digital and non-digital, in this informal, play-based learning space. Concurrently, it proposes multimodal social semiotics as an apt theoretical approach for conceptualising play as learning, and reflects upon the particular potentials and constraints offered by multimodal technologies.
LEGO House was chosen as a research setting because of its innovative uses of digital technologies in combination with traditional LEGO bricks. It can therefore be seen as a case study of multimodal technologies, offering the rich context-specific insights that case study methodologies afford [31
]. However, it is recognised that through their specificity and individuality, case studies do not easily invite generalisation between cases. For instance, the play experiences in this study are particular to LEGO House and such technologies are unlikely to be found in museums with more traditional exhibits and layout. Case studies instead invite theoretical generalisations to be made, generating relevance beyond the immediate case [32
]. In this way, multimodal social semiotics presents an analytic frame for looking at the particular digital and non-digital play experiences in LEGO House, whilst offering insights for multimodal technologies and learning more generally.
The visit to LEGO House involved a guided tour by Patrick Otley, toy designer and product development lead at LEGO, and a meeting with Amos Blanton who was involved in the development of LEGO House in his role as Research Manager. These informal discussions offered insights into LEGO House in terms of the designers’ intentions and the potential for productive exchanges between researchers and designers [33
]. Whilst The LEGO Group is a commercial business, the company’s non-profit LEGO Foundation has demonstrated a longstanding interest in forming research partnerships in order to develop, support and share evidence on play and learning. The contribution of a multimodal perspective therefore has the potential to connect with developers of toys and play-spaces whilst expanding theoretical conceptualisations, collaborative methodologies and understandings of contemporary play.
Photographs and videos of the experiences were collected during the tour of LEGO House and notes were made to record observations and discussions. These notes were elaborated in a more extended ‘research journal’ format immediately after the visits and form the basis of the examples included in this article. The three narrative accounts below can be considered research vignettes, combining description, reflection and photographs of spaces and materials. Vignettes are an apt device for recounting the research visit as they combine reflections on context with researcher interpretation [32
]. As a device typically included in case study research, vignettes support the research process as they require restructuring of thought [34
]. In this way, the examples below seek to evoke the multimodal technologies of LEGO House whilst interweaving interpretation in relation to the research aims.
What follows is an overview of LEGO House followed by examples of three play experiences that merged digital and non-digital play in innovative ways. Each is considered from a multimodal social semiotic perspective [1
] in order to examine the potentials and constraints of each experience and the consequences for play and meaning-making. The three experiences are analysed specifically in relation to the concepts of affordance
. Affordance refers to the recognition that different modes offer different potentials for making meaning [23
]. Affordance is shaped by both material properties and the mode’s social history, inevitably affecting the kinds of semiotic work a particular mode might be used for in a particular context. As modes have different affordances, multimodal social semiotics is particularly interested in the remaking of meaning across modes, encapsulated in the term ‘transduction’ [1
]. These concepts highlight that materials for play such as those made available in LEGO House have profound implications, since the affordances of one mode are different to another resulting in inevitable ‘gains and losses’ in meaning-making [27
As this research visit was limited in its duration and possible scope, this discussion focuses solely on the design of the play experiences and does not seek to examine visitors’ experiences. Further research would therefore be beneficial to build on these insights through focusing directly on analysing visitors’ interactions with the play experiences in LEGO House.
3. LEGO House: Examples of Hybrid Play Experiences
LEGO House is a 12,000-square metre building which opened in 2017 close to the LEGO headquarters in Billund, Denmark (see Figure 1
), described by the LEGO Group as ‘an experience house’ with the slogan ‘Home of the Brick’. Whilst positioning LEGO House as a playful experience for all ages, a pedagogical ethos underpins its design. This reflects the company’s strong philosophy of learning through play:
When children play they are having fun, experimenting, tinkering, messing around and making mistakes. In other words, they are learning [35
LEGO House conceptualises children’s play and learning as ‘a holistic balance of five overlapping competences’, including creativity, cognitive abilities, emotions, social play and physical play [35
]. These ‘competences’ inform the spatial design of LEGO House, arranged as five separate zones of different colours. In each zone are a number of ‘experiences’ offering opportunities to play with LEGO. The experiences are usually presented without explicit instructions or directions. Amos Blanton, who supported the development of LEGO House, explained that the concept of ‘low floors and high ceilings’ had informed the design of the play experiences, meaning they were intended to be as easily accessible but as open-ended as possible. In addition to the experience zones, LEGO House features outdoor play terraces, a gallery showcasing LEGO creations, a small library, a history collection, restaurants and a shop. Visitors are free to explore as many of the play experiences as they would like, and to spend as much time as they wish at each experience, with LEGO staff on-hand to facilitate.
Upon arrival at LEGO House each visitor is given a wristband containing a small RFID tag that can be scanned at ‘capture stations’ situated throughout the experience zones. This system enables visitors to save photographs and digital creations to an individual online profile that can be accessed through the LEGO House app after the visit. In this way, multimodal technology is embedded throughout the entire visit as a means of capturing aspects of the experiences. Technology also features in many of the experiences themselves, bringing together play with physical materials such as LEGO bricks and adding digital dimensions through use of cameras, scanners, sensors, projectors, stop-motion software and programmable robots.
What follows is a discussion of three LEGO House experiences, considering the ways in which digital and non-digital elements are combined in opportunities for play. Adopting a multimodal social semiotic perspective, the discussion focuses on the affordances
of the particular experiences and the transduction
between physical and digital modes [1
], but does not include visitors’ interactions with the experiences due to the focus of the research visit.
3.1. Character Creator
The Character Creator experience invites visitors to make their own LEGO minifigures using a vast selection of heads, torsos, legs, hairstyles and accessories provided in large low-level trays situated in one of the experience zones. The minifigure creations can then be photographed using one of the capture stations situated nearby (see Figure 2
). By placing the characters on a plinth in front of a camera, the capture station creates a snapshot of the visitor’s minifigure which is saved to the user’s wristband and can be viewed after the visit. The Character Creator capture stations also offer a choice of magazine-style overlays for the images, parodying publications such as Rolling Stone and TIME Magazine (see Figure 3
). In addition to the capture stations, the experience also features display areas where the physical minifigures can be left for other visitors to view.
In part, the capture stations seem to support an implied rule of LEGO House that visitors are expected to leave their creations behind for other users to admire, eventually to be disassembled and re-used. The capture stations provide a solution to the issue that many visitors may feel reluctant to leave their creations behind, having often invested considerable time and care in making them. The capture stations also offer an additional layer to the process of making through the possibility of adding digital overlays to the photographs, playfully drawing on conventions of particular media magazine genres. In this way, three-dimensional LEGO models become preserved as two-dimensional images digitally augmented to transform the minifigures into magazine cover stars.
Through having many components available for making minifigures, but no specified instructions or explicit goals, the importance of creativity is foregrounded in this experience and others in LEGO House. The materials both offer a direction, in that they are all elements of figures or accessories which the minifigures might hold, but invite numerous different combinations. In this way, creativity is prioritised yet also scaffolded, with prompts and the provision of materials suggesting a range of options within which the visitor has a high degree of agency.
Whilst a key potentiality of LEGO is that it can be continually assembled and reassembled, the capture stations provide a means of ‘freezing’ parts of that process so that traces of the making can be preserved and shared. In this way, moments in the ‘chain of semiosis’ [36
] typical to play with LEGO can be made visible. The capture stations therefore offer new possibilities for playful making, inviting the capturing and ongoing transformation of physical creations into digital forms. This process of transduction from physical LEGO minifigure to digital image inevitably involves both gains and losses. The capture stations can only capture what the digital camera can record, so cannot preserve sensory features of the creations such as the feel, temperature or smell of the plastic, the sound of connecting the pieces, and so on. Furthermore, as a momentary snapshot, the photos cannot depict dynamic aspects of play such as the movement of figures, nor the processes of their creation, being geared towards capturing ‘the end product’. The images created in the capture stations are therefore partial digital re-materialisations of physical creations, yet they provide insights into traces of the LEGO House play experience that would otherwise remain ephemeral.
3.2. City Architect
The City Architect experience features large interactive tables with a bustling cityscape projected onto the surface from digital projectors above (Figure 4
). This birds-eye view of a virtual city is reminiscent of the open-ended city building video game SimCity and operates on a similar premise. The implied aim of this experience is to meet the needs of animated citizens moving across the cityscape. The citizens indicate their needs through the colour of their clothing and coloured icons in small speech bubbles (e.g., a red house in a citizen’s speech bubble shows a request for more housing). Situated near the tables are colour-coded building stations where these additions to the city can be created (Figure 5
). When the constructions are placed into the square grooves on the table’s grid-like surface, the city responds with sound effects and projected animation, changing the dynamics of the city based on the colour of the base-brick that is positioned (e.g., a red brick meeting the citizens’ need for housing). If their needs are met, the animation will show citizens moving from the streets of the city to a central pre-built entertainment centre (e.g., a football stadium). When enough citizens are satisfied, a projected animation will show the city enjoying the football game, with sound effects simulating the cheering crowd. As this success requires the addition of many individually constructed elements being combined together, the experience prompts collaboration, which is further supported by the size and layout of the tables which invite many users to stand and contribute simultaneously.
City Architect therefore offers a simple city simulation, with the city’s rhythm digitally changing depending on what bricks are physically placed on the table. It is one of the more technologically complex experiences in LEGO House, making use of interactive table interfaces and projectors. Notably, it involves a departure from screen-based uses of digital technology. Instead, in the City Architect experience the projected cityscape is manipulated through the positioning of physical LEGO bricks, changing digital animations and sound effects. Multimodal perspectives have drawn attention to the profound shift in communication as the screen increasingly replaces the page as the central medium of communication, with visual modes becoming increasingly foregrounded over print [23
]. The movement away from screen-based digital interaction now brings other modes to the fore [33
]. For instance, in City Architect the gestural placement of physical, three-dimensional constructions is the prompt for two-dimensional digital animations and sound effects. As technologies continue to develop and invite further screen-less digital experiences, embodied modes, physical materials and spatial arrangements seem likely to take on newly prominent significance. The digital dimension also sustains the goal-oriented component of the play, by communicating the citizens’ needs and showing when these have been met, although there is no penalty for choosing not to complete this goal and focusing instead on the constructions themselves. In this way, the City Architect experience serves as a reminder that whatever the intended design of a particular play experience, the player’s own meaning-making in response to certain invitations, including resistance and re-design, must also be acknowledged.
Whilst creating buildings and cities might be common to ‘traditional’ play with LEGO bricks, in City Architect the digital technology is used to augment these three-dimensional structures, appearing to ‘give life’ to the city through animating inhabitants and adding sound effects. Experiences such as City Architect invite parallels with similar play in non-digital means, supporting the claim of Kress and van Leeuwen [37
] that many digital media are often a repackaging of older practices. For instance, in traditional LEGO play children might have ‘given life’ to their constructions through moving minifigures, adding their own sound effects or imagining
a bustling cityscape. In digitally augmented experiences such as City Architect, a digital realisation of this is offered in pre-designed forms. Whilst on the one hand such technologies may offer a ‘magical’ dimension to the experience of construction, adding digital layers onto a traditional play experience, it raises questions regarding the place of imagination in play. Future studies that explore the merging of digital and non-digital aspects of play might turn to children themselves to explore the issue of imagination further, exploring the extent to which children resist or re-make designers’ intentions for play experiences.
3.3. Fish Designer
The Fish Designer experience invites visitors to make a fish using a selection of coloured LEGO bricks (Figure 6
A) then scan and convert their fish design into a digital form by placing the construction on a capture station platform (Figure 6
B–D). Once scanned, visitors can add eyes and a mouth to their fish (Figure 6
E) and watch it become animated in a digital aquarium alongside other visitors’ fish. Through customising the eyes and mouth, fish will display different personality traits as they swim together and respond to aquarium animations such as sharks. As new fish populate the tank, older fish will gradually fade into the distance of the aquarium and eventually disappear, whilst the capture stations save an image of the digital fish to the user’s wristband to be accessed via the app after the visit (Figure 6
F). Near to the capture stations and screens is a display area where the physical LEGO brick fish can also be left on display.
Use of sound and animation at the capture stations helps sustain an illusion that the physical LEGO brick fish is becoming digital. For instance, through an on-screen ‘scanning’ animation and sound effect, and through animation showing the fish swimming from the small capture station screen into the large aquarium screen through a connecting tube (see Figure 7
). In this way, the process of transduction between a three-dimensional object, two-dimensional image and two-dimensional animation is brought to the fore. The process of re-making becomes the main feature of the experience, playing with the idea that a brick construction has been ‘brought to life’. The multimodal technology draws upon affordances of digital moving image to give the appearance of ‘life’ to an inanimate construction through moving image and sound. The experience also plays with time and space, allowing visitors’ fish to exist in the same space simultaneously despite being made at different times, and to live on after the physical blocks have been disassembled. In this way, the experience highlights the potential of digital technologies to support opportunities for play across physical, temporal and spatial boundaries.
However, once the digital fish is released into the aquarium it is out of the control of its designer. It can be watched as it interacts with other fish but can no longer be played with as the physical LEGO brick fish might. The Fish Designer therefore highlights again the gains and losses inherent in play experiences which involve movement between physical and digital forms. On the one hand, the experience creates a re-materialisation of a brick fish that takes on life-like animate qualities and can add new spatial and temporal dimensions to play. Yet it also demonstrates limitations of the form and the process of transduction when a physical object is re-made digitally.
LEGO House resists easy definition, combining elements of a museum, gallery, studio and playground, and highlights the close connections between activities such as playing, tinkering, designing, making and learning. In this way, LEGO House might be considered an example of a ‘third space’, occupying a position that is neither home nor school, instead existing as a ‘makerspace’ in which playful opportunities for creating are supported in both digital and non-digital forms. LEGO House is a distinctive example as it merges physical LEGO bricks with digital technologies in new and innovative ways, whilst also drawing on many traditional play practices [37
]. Its invitations to creatively combine and move between digital and non-digital forms offer new opportunities for meaning-making, whilst simultaneously highlighting the gains and losses that such movement entails.
In light of the aims of the research visit, this article proposes that multimodal social semiotics is an apt theoretical
perspective to explore hybrid contexts for meaning-making, offering a balanced insight into digital and non-digital dimensions of play and critically examining what such designs make possible, whilst avoiding the common extremes of moral panics or ‘technotopian’ ideals often brought into discussions about young children’s uses of technology. This article therefore extends existing multimodal research into children’s play and toys [23
] by looking at the particular affordances brought about by play experiences that merge digital and non-digital elements.
This article suggests there are particular methodological possibilities for collaborative research-design partnerships regarding digital and non-digital play. Companies such as LEGO Group have extensive experience and expertise in designing innovative toys and play-spaces, whilst multimodal social semiotic research is able to bring into focus the particular potentials and constraints for meaning-making. Further collaborative research may potentially reify the affordances of digital and non-digital play experiences as these continue to change, as although LEGO House is currently a cutting-edge development for the LEGO Group, technology continues to develop apace. It will be interesting to see the extent to which LEGO update or replace their experiences to reflect new technological developments. For instance, as ‘virtual reality’ and ‘augmented reality’ continue to become more prominent, the potential to further blur the digital and non-digital is likely to invite further possibilities for play experiences. Additionally, whilst the current play experiences focus on turning the physical into the digital (e.g., the capture stations and Fish Designer), the move from digital into non-digital is not yet an aspect explored in the LEGO House. With technologies such as 3D printers and laser cutters increasingly inviting movement from digital into physical forms in makerspaces such as FabLabs, the possibility to design a creation digitally and have it physically realised in LEGO may be one future possibility. Another may be to find further ways to sustain play experiences after the visit through the LEGO House app, for instance through enabling digital creations such as the fish to become animated virtual pets on mobile devices. This article highlights that there will be an ongoing need to evaluate LEGO House play experiences, and makerspaces more generally, in light of technological innovation, and that there are productive synergies to be realised by combining the perspectives of designers and insights from researchers to give careful consideration to new possibilities for play.
Through closely examining the LEGO House examples above, this article highlights some practical ways in which digital and non-digital play is supported in third-space settings. Whilst this article focuses on the design of LEGO House experiences from a multimodal perspective, further research could explore the ways in which visitors interact with these experiences, and how individuals bring their own agency and interest to the intended designs for play. Such research would support insights into visitors’ experience of LEGO House and would enable attention to meaning-making and signs of learning in this distinctive and innovative context.
LEGO House can be seen as part of a growing recognition of the power of play, both in its own right and in relation to learning. Through a playful pedagogical ethos underpinning its design, LEGO House acknowledges that deeply important learning can happen in informal times and places that are not positioned as direct acts of teaching. In this way, LEGO House contributes to ongoing discussions surrounding the educational approaches, spaces and materials that are necessary for supporting playful meaning-making. Specifically, it considers the role of the digital in such an approach, and through its embedded use of digital technologies alongside non-digital forms, can be seen to encapsulate the ‘hundred languages’ philosophy of the preschools of Reggio Emilia in which digital technologies are valued as means of amplifying and multiplying meaning-making (see references [19
]). Whilst difficult to define, in many ways LEGO House might be best described as an atelier or studio, and connects to the rise in ‘maker’ culture and particular interest in ‘digital making’ [38
]. In this way, experiences such as those found in LEGO House might contribute towards approaches to teaching and learning which support children to develop the playfully creative skills and knowledge required for the digital age. Making evident the opportunities for meaning-making that exist in informal, play-based settings such as LEGO House therefore has the potential to recognise and give value to playful meaning-making in both digital and non-digital forms which may otherwise be taken for granted or go unnoticed.