Abstract
As systems, for example vehicle systems, get increasingly autonomous and information intense, the information exchanged with the user, i.e., the operator, are increasingly becoming a designed interaction. This work investigates how interaction technologies, interaction design principles, and machine information systems can be used to provide user experiences and efficient interaction between the operator and industrial mobile machines; for example, agricultural machines and construction machines. In this pursuit the research aims to explore the use of mixed reality interaction and visual presentation using see-through interfaces and symbolic metaphors, to enhance the interaction for operators working with these types of machines.
1. Introduction and Motivation for Research
Interaction design is increasingly crucial as an ingredient in product development and digitalization. The need is nourished by a trend where software based functionality is becoming increasingly important in all types of product features. Where more information is created and where new technology moves the frontier for where interaction between human and computer takes place. Furthermore, as machines get connected, even more information will be exchanged with and between machines as well as their operators. With higher levels of autonomy, it is also likely that the activities and purpose of the operator will transform more towards managerial activities than operational. This affecting the interaction with the machine and the need for information.
New information based systems can aid the operator’s task fulfillment, but there is also a risk that information is added that negatively affect the mental load and attention of the operator. This potentially increasing the risk of failure and human safety [1]. One example is the Llanbadarn Automatic Barrier incident report where a train passed a crossing with the bars raised. One reason being that the operator was occupied with the driver machine interface and therefore missed the crossing indicator [2].
Utilization of information and interaction design in these types of machines can also positively impact a sustainable society. For example through less fuel consumption due to optimized operation and navigation. Or in agriculture, where use of pesticides and fertilizing can be more precise and minimized, due to sensory information, big data analysis and information support to the operator. A good use of sensors, information, and interaction can also improve situational awareness and increase operators wellbeing and sustained operation. For example by reducing the excessive movements made by operators to cover angles with limited sight [3]. Interaction design can also improve information detection and intake. For example when the information is mediated within the visual attention area, thus minimizing the time required to refocus on graphical displays in the cabin and then back into the surroundings [4].
The aim of this research is to explore interaction design with heavy vehicles working in information intensive applications, such as, precision farming and construction machines. The research combines software engineering and interaction design together with an industrial perspective.
2. A Description of the Proposed Approach/Solution
The approach for this project going forward is to practically explore interaction design and technology, by production of visualization concepts and prototypes. One specific approach is to explore the use of mixed reality to present information on the cabin windows, in the operator’s line of sight. This way the user doesn’t have to look away from the operation area to acquire system information. This would also provide a larger screen area for information presentation.
Using mixed reality interaction, thus blending system information with the real world, have potential to enhance situational awareness. Following is a short imaginative scenario to illustrate this:
The operator of an excavator prepares a ground on a building site. Visible in front of the operator are the different leveling areas and their needed adjustment, based on the blueprints and scanning of the surrounding. The system also presents and alerts about pipes and electrical lines in the soil. The operator can hence focus attention on the area of operation, instead diverting attention on to display based precision systems. Meanwhile, a supporting autonomous dump truck is approaching the excavator. It’s carrying additional gravel to fill an area. Upon arrival the excavator system displays an alert in the visual range of the operator, indicating close range movement. The planned passage and unloading place for the dump truck is presented and the operator can acknowledge the passage.
This example can be extended with more information, for example, pedestrian information, sensory data, full 3D visualization of the property construction etc. However, there is a risk that the operator will be flooded with information. Current visual technologies for full window visualization also lack a forward positioned focal depth point, resulting in a constant visual re-focus for the operator. Also, making full immersion with the surrounding require a lot of information and sensory integration, affecting complexity, cost, robustness, and possibility of safety certification.
The research thus seeks a minimalistic approach of presentation, including a common language of interaction. This language could be a uniform symbol language that can be used in many types of applications in several industrial machinery contexts and handle diverse levels of criticality.
3. A Summary of Background and Related Work
Different types of transparent interaction systems and augmented interfaces, that keep the user visual attention in line with operation, is currently an area of state-of-the-art interest [5], empowered by higher embedded computation power and increased availability of visualization technology. The field itself isn’t new, mixed reality in terms of Head-Up Displays (HUDs) have been used for a long time in specific areas, aerospace being one early adopter. The technology has recently become increasingly common in automotive applications, in particular cars, where it is being extensively researched. However, the information needed in industrial applications is different than in cars, as it isn’t only about transportation, but also about the production process performed. Also, the size of HUD projectors used in cars, including their visual display area, is not suitable for heavy machinery.
Mixed reality has also been evaluated in the heavy machinery industry, indicating possible benefits in ergonomics, information intake and productivity [6,7]. These tests have mainly focused on taking production information currently visualized on displays, and replicating this in the field of view of the operator. Thus focusing less on renewed ways to present production information, as well as mediation of additional information.
Another alternative is to use head-worn mixed reality displays [8]. Together with digital models, this technology is maturing to replace blueprints while building properties [9]. But these devices still have limited battery time and the operator have to take extra steps by putting on equipment before being able to operate or move in and out of the vehicle. Mobile devices (smart phones) can also be used in augmented solutions, for example, in mobile mixed reality interfaces where information is overlaid on the display of the device [10,11]. But these solutions limits operation, when the users hands are occupied holding the device. Still, it can be a good complement when the operator is outside of the vehicle, or for people that aren’t actively working hands-on.
Additionally, the way to mediate information in information intense and increasingly autonomous vehicles is an area of research. In farming, Sørensen et.al mention that acquisition and analysis of information still proves a demanding task [12]. Furthermore, the availability of data does not warrant the understanding or usefulness of the data to the user [13]. There are also challenges with information availability, where many industrial segment, manufacturers and system providers lack open access to information necessary to build integrated systems. Operation can also be in places with limited communication infrastructure, resulting in limited data exchange.
4. A Brief Discussion on the Applied Research Methodology, Including how the Solution Is Going to be Evaluated
Interaction design research can be performed through design practice [14,15,16]. Fällman describes Interaction Design in the activities of Study, Practice, and Exploration [17].
The majority of the initial work in this research project have been on understanding, through study and observation. A strong concept in the continued work is to explore possibilities through practicing design and reflection on possible future outcomes [16,18]. The result of the practice can be expressed in many forms, for example, sketches [19], through artefacts and systems [20].
Evaluation of the concepts and prototypes will mainly be made through simulators and qualitative interviews, as access to real working vehicles is a limited source. For mature concepts, there is an aim to do quantitative user evaluation with a bigger sample set.
5. Expected Contributions, and Results so Far
The initial work [21] has, among others, resulted in:
- Means to do efficient operator’s studies, with minimal interruption of work, using eye tracking. This work included a qualitative study of operator’s attention areas in different vehicle types [4].
- Studies of the role of user understanding in the creation of autonomous vehicles [22].
- Possible use of mixed reality interfaces in the lifetime of an industrial vehicle [23].
- Realization of a mixed reality simulator with free head movement [24].
The plan ahead is to use this experience in the second phase of the research, going into design explorations of possible interaction design concepts. In sum, the expectation is that this will contribute to the understanding of information exchange between software intensive heavy vehicles and its operators, its scenarios and technologies, its use, its challenges, and possibilities.
Conflicts of Interest
The author declares no conflict of interest.
References
- Lauber, F.; Follmann, A.; Butz, A. What You See Is What You Touch: Visualizing Touch Screen Interaction in the Head-Up Display. In Proceedings of the 2014 Conference on Designing Interactive Systems, Vancouver, BC, Canada, 21–25 June 2014; pp. 171–180. [Google Scholar]
- Department for Transport. Incident at Llanbadarn Automatic Barrier Crossing (Locally Monitored), near Aberystwyth, 19 June 2011; Department for Transport: London, UK, 2012.
- Eger, T.; Godwin, A.; Henry, D.J.; Grenier, S.G.; Callaghan, J.; Demerchant, A. Why vehicle design matters: Exploring the link between line-of-sight, driving posture and risk factors for injury. Work 2010, 35, 27–37. [Google Scholar] [PubMed]
- Wallmyr, M. Seeing through the eyes of off-highway vehicle operators. In (Accepted at) The 16th IFIP TC.13 International Conference on Human-Computer Interaction, INTERACT ’17; Springer: Mumbai, India, 2017; p. 21. [Google Scholar]
- Betancur, J.A.; Villa-Espinal, J.; Osorio-Gómez, G.; Cuéllar, S.; Suárez, D. Research Topics and Implementation Trends on Automotive Head-Up Display Systems. 2016. Available online: https://link.springer.com/journal/12008/onlineFirst/page/4 (accessed on 8 June 2017).
- Nordlie, A.; Till, S. Head-Mounted Displays for Harvester Operators—A Pilot Study; KTH Royal Institute of Technology: Stockholm, Sweden, 2015. [Google Scholar]
- Lagnel, O.; Engstr, J. Bättre Arbetsmiljö med Head Up Display; JTI: Geneva, Switzerland, 2015. [Google Scholar]
- Kress, B.; Starner, T. A review of head-mounted displays (HMD) technologies and applications for consumer electronics. In Proceedings of the Photonic Applications for Aerospace, Commercial, and Harsh Environments IV (SPIE 8720), Baltimore, MD, USA, 29 April 2013. [Google Scholar]
- Nohrstedt, L. Masken Styr Armeringen på ESS-Bygget. Available online: http://www.nyteknik.se/bygg/masken-styr-armeringen-pa-ess-bygget-6840540 (accessed on 20 May 2017).
- Van Krevelen, R.; Poelman, R. A survey of augmented reality technologies, applications and limitations. Int. J. Virtual Real. 2010, 9, 1–20. [Google Scholar]
- Seichter, H.; Grubert, J.; Langlotz, T. Designing mobile augmented reality. In Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’13), Munich, Germany, 27–30 August 2013; p. 616. [Google Scholar]
- Sørensen, C.G.; Fountas, S.; Nash, E.; Pesonen, L.; Bochtis, D.; Pedersen, S.M.; Basso, B.; Blackmore, S.B. Conceptual model of a future farm management information system. Comput. Electron. Agric. 2010, 72, 37–47. [Google Scholar] [CrossRef]
- Chinthammit, W.; Duh, H.B.-L.; Rekimoto, J. HCI in food product innovation. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI EA ’14), Toronto, ON, Canada, 26 April–1 May 2014; pp. 1111–1114. [Google Scholar]
- Zimmerman, J.; Forlizzi, J.; Evenson, S. Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), San Jose, CA, USA, 30 April–3 May 2007; p. 493. [Google Scholar]
- Höök, K.; Löwgren, J. Strong concepts. ACM Trans. Comput. Interact. 2012, 19, 1–18. [Google Scholar] [CrossRef]
- Schön, D. The Reflective Practitioner: How Professionals Think in Action; Basic Books: New York, NY, USA, 1983. [Google Scholar]
- Fallman, D. The Interaction Design Research Triangle of Design Practice, Design Studies, and Design Exploration. Des. Issues 2008, 24, 4–18. [Google Scholar] [CrossRef]
- Bardzell, J. Interaction criticism: An introduction to the practice. Interact. Comput. 2011, 23, 604–621. [Google Scholar] [CrossRef]
- Buxton, B. Sketching User Experiences; Morgan Kaufman: Burlington, MA, USA, 2007. [Google Scholar]
- Cross, N. Design Research: A Disciplined Conversation. Des. Issues 1999, 15, 5–10. [Google Scholar] [CrossRef]
- Wallmyr, M. Exploring Interaction Design Perspectives on Heavy Vehicles; Mälardalen University: Västerås, Sweden, 2017. [Google Scholar]
- Wallmyr, M. Understanding the user in self-managing systems. In Proceedings of the 2015 European Conference on Software Architecture Workshops (ECSAW ’15), Dubrovnik, Croatia, 7–11 September 2015; pp. 1–4. [Google Scholar]
- Wallmyr, M. Interactions and Applications for See-Through Technologies—Industrial application examples. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI ’14), Helsingfors, Finland, 26–30 October 2014; pp. 793–796. [Google Scholar]
- Kade, D.; Wallmyr, M.; Holstein, T.; Lindell, R.; Ürey, H.; Özcan, O. Low-Cost Mixed Reality Simulator for Industrial Vehicle Environments. In Proceedings of the 8th International Conference, VAMR 2016, Held as Part of HCI International 2016, Toronto, ON, Canada, 17–22 July 2016; Volume 9740, pp. 597–608. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2017 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).