Exploring Interaction Design with Information Intense Heavy Vehicles ^†

Abstract

As systems, for example vehicle systems, get increasingly autonomous and information intense, the information exchanged with the user, i.e., the operator, are increasingly becoming a designed interaction. This work investigates how interaction technologies, interaction design principles, and machine information systems can be used to provide user experiences and efficient interaction between the operator and industrial mobile machines; for example, agricultural machines and construction machines. In this pursuit the research aims to explore the use of mixed reality interaction and visual presentation using see-through interfaces and symbolic metaphors, to enhance the interaction for operators working with these types of machines.

1. Introduction and Motivation for Research

Interaction design is increasingly crucial as an ingredient in product development and digitalization. The need is nourished by a trend where software based functionality is becoming increasingly important in all types of product features. Where more information is created and where new technology moves the frontier for where interaction between human and computer takes place. Furthermore, as machines get connected, even more information will be exchanged with and between machines as well as their operators. With higher levels of autonomy, it is also likely that the activities and purpose of the operator will transform more towards managerial activities than operational. This affecting the interaction with the machine and the need for information.

New information based systems can aid the operator’s task fulfillment, but there is also a risk that information is added that negatively affect the mental load and attention of the operator. This potentially increasing the risk of failure and human safety [1]. One example is the Llanbadarn Automatic Barrier incident report where a train passed a crossing with the bars raised. One reason being that the operator was occupied with the driver machine interface and therefore missed the crossing indicator [2].

Utilization of information and interaction design in these types of machines can also positively impact a sustainable society. For example through less fuel consumption due to optimized operation and navigation. Or in agriculture, where use of pesticides and fertilizing can be more precise and minimized, due to sensory information, big data analysis and information support to the operator. A good use of sensors, information, and interaction can also improve situational awareness and increase operators wellbeing and sustained operation. For example by reducing the excessive movements made by operators to cover angles with limited sight [3]. Interaction design can also improve information detection and intake. For example when the information is mediated within the visual attention area, thus minimizing the time required to refocus on graphical displays in the cabin and then back into the surroundings [4].

The aim of this research is to explore interaction design with heavy vehicles working in information intensive applications, such as, precision farming and construction machines. The research combines software engineering and interaction design together with an industrial perspective.

2. A Description of the Proposed Approach/Solution

The approach for this project going forward is to practically explore interaction design and technology, by production of visualization concepts and prototypes. One specific approach is to explore the use of mixed reality to present information on the cabin windows, in the operator’s line of sight. This way the user doesn’t have to look away from the operation area to acquire system information. This would also provide a larger screen area for information presentation.

Using mixed reality interaction, thus blending system information with the real world, have potential to enhance situational awareness. Following is a short imaginative scenario to illustrate this:

The operator of an excavator prepares a ground on a building site. Visible in front of the operator are the different leveling areas and their needed adjustment, based on the blueprints and scanning of the surrounding. The system also presents and alerts about pipes and electrical lines in the soil. The operator can hence focus attention on the area of operation, instead diverting attention on to display based precision systems. Meanwhile, a supporting autonomous dump truck is approaching the excavator. It’s carrying additional gravel to fill an area. Upon arrival the excavator system displays an alert in the visual range of the operator, indicating close range movement. The planned passage and unloading place for the dump truck is presented and the operator can acknowledge the passage.

This example can be extended with more information, for example, pedestrian information, sensory data, full 3D visualization of the property construction etc. However, there is a risk that the operator will be flooded with information. Current visual technologies for full window visualization also lack a forward positioned focal depth point, resulting in a constant visual re-focus for the operator. Also, making full immersion with the surrounding require a lot of information and sensory integration, affecting complexity, cost, robustness, and possibility of safety certification.

The research thus seeks a minimalistic approach of presentation, including a common language of interaction. This language could be a uniform symbol language that can be used in many types of applications in several industrial machinery contexts and handle diverse levels of criticality.

3. A Summary of Background and Related Work

Different types of transparent interaction systems and augmented interfaces, that keep the user visual attention in line with operation, is currently an area of state-of-the-art interest [5], empowered by higher embedded computation power and increased availability of visualization technology. The field itself isn’t new, mixed reality in terms of Head-Up Displays (HUDs) have been used for a long time in specific areas, aerospace being one early adopter. The technology has recently become increasingly common in automotive applications, in particular cars, where it is being extensively researched. However, the information needed in industrial applications is different than in cars, as it isn’t only about transportation, but also about the production process performed. Also, the size of HUD projectors used in cars, including their visual display area, is not suitable for heavy machinery.

Mixed reality has also been evaluated in the heavy machinery industry, indicating possible benefits in ergonomics, information intake and productivity [6,7]. These tests have mainly focused on taking production information currently visualized on displays, and replicating this in the field of view of the operator. Thus focusing less on renewed ways to present production information, as well as mediation of additional information.

Another alternative is to use head-worn mixed reality displays [8]. Together with digital models, this technology is maturing to replace blueprints while building properties [9]. But these devices still have limited battery time and the operator have to take extra steps by putting on equipment before being able to operate or move in and out of the vehicle. Mobile devices (smart phones) can also be used in augmented solutions, for example, in mobile mixed reality interfaces where information is overlaid on the display of the device [10,11]. But these solutions limits operation, when the users hands are occupied holding the device. Still, it can be a good complement when the operator is outside of the vehicle, or for people that aren’t actively working hands-on.

Additionally, the way to mediate information in information intense and increasingly autonomous vehicles is an area of research. In farming, Sørensen et.al mention that acquisition and analysis of information still proves a demanding task [12]. Furthermore, the availability of data does not warrant the understanding or usefulness of the data to the user [13]. There are also challenges with information availability, where many industrial segment, manufacturers and system providers lack open access to information necessary to build integrated systems. Operation can also be in places with limited communication infrastructure, resulting in limited data exchange.

4. A Brief Discussion on the Applied Research Methodology, Including how the Solution Is Going to be Evaluated

Interaction design research can be performed through design practice [14,15,16]. Fällman describes Interaction Design in the activities of Study, Practice, and Exploration [17].

The majority of the initial work in this research project have been on understanding, through study and observation. A strong concept in the continued work is to explore possibilities through practicing design and reflection on possible future outcomes [16,18]. The result of the practice can be expressed in many forms, for example, sketches [19], through artefacts and systems [20].

Evaluation of the concepts and prototypes will mainly be made through simulators and qualitative interviews, as access to real working vehicles is a limited source. For mature concepts, there is an aim to do quantitative user evaluation with a bigger sample set.

5. Expected Contributions, and Results so Far

The initial work [21] has, among others, resulted in:

• Means to do efficient operator’s studies, with minimal interruption of work, using eye tracking. This work included a qualitative study of operator’s attention areas in different vehicle types [4].
• Studies of the role of user understanding in the creation of autonomous vehicles [22].
• Possible use of mixed reality interfaces in the lifetime of an industrial vehicle [23].
• Realization of a mixed reality simulator with free head movement [24].

The plan ahead is to use this experience in the second phase of the research, going into design explorations of possible interaction design concepts. In sum, the expectation is that this will contribute to the understanding of information exchange between software intensive heavy vehicles and its operators, its scenarios and technologies, its use, its challenges, and possibilities.