Conceptual Design and Evaluation of Windshield Displays for Excavators

Abstract

This paper investigates the possible visualization using transparent displays, which could be placed on the excavator’s windshield. This way, the information could be presented closer to operators’ line of sight, without fully obstructing their view. Therefore, excavator operators could acquire the supportive information provided by the machine without diverting their attention from operational areas. To ensure that there is a match between the supportive information and operators’ contextual needs, we conducted four different activities as parts of our design process. Firstly, we looked at four relevant safety guidelines to determine which information is essential to perform safe operations. Secondly, we reviewed all commercially available technologies to discover their suitability in the excavator context. Thirdly, we conducted a design workshop to generate ideas on how the essential information should look like and behave based on the performed operation and the chosen available technology. Fourthly, we interviewed seven excavator operators to test their understanding and obtain their feedback on the proposed visualization concepts. The results indicated that four out of six visualization concepts that we proposed could be understood easily by the operators and we also revised them to better suit the operators’ way of thinking. All the operators also positively perceived this approach, since all of them included at least three visualization concepts to be presented on the windshield.

1. Introduction

Excavators are versatile machines that can be used for various purposes, for example, lifting, drilling, demolishing, and cutting, depending on the attachment being used. Due to their versatility, excavators are not only used in the construction sector, but also in other sectors, such as mining and forestry. Operating an excavator is a cognitively demanding task [1], as operators are required to control the machine, observe the surroundings, and possibly cooperate with nearby ground workers or machines. Kazan and Usmen [2] compared the accident data that involved four types of heavy machinery in the U.S. and they found that excavators were involved in 24.8% of 1200 reported accidents, where half of those excavator-related accidents led to fatalities. The operator’s misjudgment during hazardous situations was also cited as the most common cause of those accidents.

Modern excavators are increasingly equipped with information systems that provide supportive information to operators [3]. The supportive information is mainly presented using head-down displays, which are installed inside the cabin [4]. However, our prior research suggested that operators generally pay little attention to the presented information [5], as the head-down display is usually placed far from operational areas to avoid obstructing operators’ view (see Figure 1 for an example). This situation is not desirable, since operators are not fully aware of the information shown on the head-down display and they may not be able to act accordingly in hazardous situations. In addition, the benefit of having the supportive information is diminished if operators do not utilize it.

In this paper, we hypothesize that it would be beneficial for operators to have the supportive information presented near their line of sight, for example, on the windshield as they spend significant amount of time looking through it [5]. Therefore, operators could acquire the supportive information without diverting their attention from operational areas. The idea of presenting information on the windshield is not new at all, as it could be traced back to fighter planes in the 1970s [6], and since then the idea has been progressively adopted in the automotive domain [7]. In the heavy machinery domain, this approach led to mixed findings, where some studies suggested improved performance and some others suggested insignificant differences (see Sitompul and Wallmyr [8] for the review). Although there could be various reasons that influenced the participants’ overall performance, we found from the review that operators were rarely consulted when designing the proposed visualization systems, since their involvement was limited to evaluating the visualization systems that have been developed. Therefore, it is still unclear to what extent that there was a match between the presented information and the operator’s contextual needs in the visualization systems that have been proposed so far. This aspect is relevant to be addressed, since the match between the presented information and the performed task possibly reduces users’ cognitive workload and improves their performance [9,10], since less cognitive resources are spent to process the information and perform the required task.

In this paper, we focused on investigating how to design the information visualization on the excavator’s windshield that fits to operators’ contextual needs. In other words, we need to determine which information is needed by operators to operate the machine safely, when is the right time to present the information, where the information should be presented, and how the information should be visualized [11,12]. This approach would possibly prevent distracting and obstructing operators with less suitable information. To address these requirements, we have defined four research questions in this study:

RQ1.

What kind of information do operators need to know in order to perform safe operations?

RQ2.

What commercially available technologies can be used to present information near line of sight and what are their limitations?

RQ3.

How should the supportive information be visualized based on the chosen technology and the currently performed operation?

RQ4.

How do excavator operators perceive the proposed visualization approach?

2. Related Work

Considering the limitation of head-down displays, many studies have investigated new ways for visualizing information in heavy machinery, including excavators. The proposed approaches varied from one study to another, but we could see that presenting information on the windshield and projecting information directly on the object of interest as the most common approaches (see Sitompul and Wallmyr [8] for the review). Specifically for excavators, Akyeampong et al. [13] found that presenting the information on the windshield reduced mental workload compared to when the information was displayed in lower locations (see the left image in Figure 2). Based on our prior study [4], we also found that the information acquisition rate was higher when the information was projected directly on the windshield compared to when a head-down display was used (see the right image in Figure 2).

Similar to the two studies mentioned above, most of the studies in the heavy machinery domain were done in simulated environments, thus it is still unclear to what extent the proposed approaches would work in real machines and real operations. Nevertheless, there were six studies that built the physical visualization systems, which could be deployed in real machines. In the context of mobile cranes, Fang et al. [14] investigated the use of a bigger head-down display, where the operator could see multiple views around the machine and the supportive information. In the context of tractors, Fernandez et al. [15] explored the use of a head-mounted display that showed navigational information and which areas that have been treated by the tractor. Still in the context of tractors, Palonen et al. [16] also used a head-mounted display, but they probed on how to visualize the environment that was occluded by the machine’s part, thus operators could observe blind spots around the machine. Finally, there were three studies that explored the cabin’s windshield as a place to visualize the information. In the context of snowplow trucks, Rakauskas et al. [17] used a projection head-up display for indicating which areas that should be cleared from snow. Englund et al. [18] also used a projection head-up display to present bucking-related information on the forest harvester’s windshield. In the context of off-shore cranes, Kvalberg [19] considered using a transparent display, which is the display that emits its own light and becomes transparent when the light is off, for presenting the lifting load of an off-shore crane.

Only half of the aforementioned studies reported some kind of user evaluations, as the remaining studies were limited to technical evaluations only. Fang et al. [14] found that the operators required less time to complete the given task, had more correct responses, and produced higher level of situation awareness when using the proposed system, compared to when no support was provided. Yet, the operators still commented that the bigger head-down display was still considered too small and the presence of the display could also obstruct their view. Rakauskas et al. [17] found that the operators drove the snowplow truck more slowly, controlled the steer more frequently, and had higher mental workload, while using the projection head-up display. On the contrary, Englund et al. [18] found that there was no significant difference on operators’ performance when they were using the projection head-up display or the head-down display. There could be many reasons that influenced these different results, such as prior experience, which information that was presented, how the information was visualized, trust on new visualization systems, etc. However, based on the review [8], we found that the proposed visualization systems were mainly designed by the researchers and operators were rarely consulted. In the studies that involved real operators, their involvement was limited to the evaluation of physical visualization systems that have been developed. Therefore, it is still unclear to what extent the presented information matched the operators’ contextual needs. Taking this situation into account, there is a need for a study that proposes visualization systems based on the match between the presented information and operators’ contextual needs [9].

3. Methods

To address the research questions mentioned in Section 1, four different activities were conducted, as described in the following subsections.

3.1. Using Safety Guidelines to Find which Information Is Important for Safe Operations

To find the answer for “RQ1—What kind of information do operators need to know in order to perform safe operations?”, we used four safety guidelines [20,21,22,23] as the source of information on how to operate excavators safely. This information could also be collected by asking operators or domain experts. However, this approach may not be effective, since operators may have different operational preferences or requirements. On the other hand, the safety guidelines describe things that are applicable to all operators, regardless of different preferences. The safety guidelines do not only describe safety requirements for operators, but also for supervisors, maintenance workers, or ground workers, as well as safety requirements for before, during, and after the operation. However, in this study, we specifically focused on what excavator operators should know or do during the operation.

After reviewing the safety guidelines, we found that the requirements are mainly provided to prevent two kinds of incidents: collision and loss of balance. To prevent any collision, operators must know the presence of nearby ground workers, existing structures, or other machines that may exist, as well as be fully aware on what the excavator is about to do. To avoid the excavator from collapsing due to the loss of balance, which could occur due to various factors, such as unstable ground, steep ground, or lifting an excessive load, operators must know the current state of the machine and never operate the machine beyond the permissible conditions.

3.2. Reviewing Commercially Available Technologies for Presenting Information Near Line of Sight

To address “RQ2—What commercially available technologies can be used to present information near line of sight and what are their limitations?”, we reviewed the available technologies to discover their advantages and disadvantages in the heavy machinery context. Technologies play a vital role here, since they highly influence how the information could be formed and visualized to the user. We also used the lessons learned from prior studies in Section 2 that deployed the physical visualization systems in real machines and tested them with operators as an additional input on what would and would not work in practice.

After conducting the review, we found that all commercially available technologies have some advantages and disadvantages. Using head-mounted displays allows operators to see the supportive information exactly within their line of sight. However, this option could be ergonomically uncomfortable for operators, since they are also required to wear protective helmets [23]. Although newer head-mounted displays come with improved ergonomics, they are still uncomfortable to be worn for long hours [24]. Both projection head-up displays and transparent displays are two alternatives that could be considered, as operators could see the supportive information without wearing additional equipment. The main drawback of projection head-up displays is the projected information may deteriorate in bright environments [25]. Rakauskas et al. [17] had to conduct their experiment in the night to specifically avoid this issue. Englund et al. [18] carried out their experiment in the daytime and the operators commented that the contrast between the projected image and the environment was low, which made the presented information difficult to see. Unlike projection head-up displays, transparent displays are strong against external light, since they emit their own light instead of using reflection [26]. However, the information that could be visualized is fixed to what has been specified when the display is manufactured and the design cannot be modified later. Furthermore, current transparent displays are also limited in terms of colors, since only green and yellow are currently available. Here, we did not consider bigger head-down displays, like what Fang et al. [14] used, since the physical presence of the display could obstruct operators’ view.

Taking into account both advantages and disadvantages of all commercially available technologies and the lessons learned from prior studies, we were interested to further explore how we could present supportive information to excavator operators using transparent displays. In addition, the use of transparent displays in the heavy machinery domain has also not been explored much in the literature, since what Kvalberg [19] has done was limited to the technical evaluation (see Figure 3). This choice also served as the boundary for our design space, since the type of displays highly influence how the information could be formed and visualized.

3.3. Conducting a Design Workshop to Generate Visualization Ideas

Regarding “RQ3—How should the supportive information be visualized based on the chosen technology and the currently performed operation?”, we approached this question through a design workshop, where three of us generated visualization ideas. We set the boundary for the design space based on the findings from the safety guidelines, thus we did not generate design visualization ideas beyond preventing collision and loss of balance. Through sketching, we explored a plethora of visualization ideas that could help excavator operators avoiding collisions with surrounding objects and maintaining the balance of the machine (see Figure 4). At the same time, we also scrutinized the sketches based on their suitability for representing the performed situation. As the chosen technology, transparent displays also bring some constraints on how the information can be designed and visualized. Transparent displays have a limited number of segments, where each segment can be either illuminated or not, and the illuminated segments should visually represent something. The segments in transparent displays can take different shapes, but their positions cannot be changed once the display has been manufactured [26]. We then selected some of the visualization concepts and refined their visual appearances and behaviors based on these design constraints.

As the result of the design workshop, we came up with visualization concepts for proximity warning, balance-related information, undercarriage direction, relative load capacity, and warning sign that suit both appearance and capability of transparent displays. Note that the different kinds of information mentioned above could be made available using existing technologies in modern excavators. The only exception is the proximity warning, since it is, to some extent, still under research and there are various ways that have been proposed so far (see Jo et al. [3] for the review). In this study, we assumed that the proximity information could be somehow made available using any kind of sensors, and thus we focused on how the information should be visualized based on transparent displays. As mentioned in Section 2, current transparent displays are still limited to green or yellow, thus we based our concepts with yellow color as this color is more appropriate to represent a warning. The description for each visualization concept is presented below.

3.3.1. Proximity Warning

The proximity warning concept was made based on three circles that indicate three different distances from the excavator to the obstacle (see the left image in Figure 5). In this concept, there are two groups of segments that each represents the presence of obstacle(s) on the left side or the right side of the machine (see the right image in Figure 5). The visualization in this concept was always based on to the direction where the cabin is facing. The left segments will be turned on when there is an obstacle on the left side of the machine, and vice versa. The curved segments show the position of the obstacle and its proximity to the machine. The horizontal segments in the middle indicate the altitude of the obstacle with respect to the excavator. The images in Figure 6 illustrate some scenarios that use the proximity warning concept. Note that here we used humans as the obstacle for simplification purposes. In practice, the obstacle can also be structures, such as buildings or overhead power lines.

3.3.2. Balance-Related Information

From the safety guidelines, we learned that the center of gravity of an excavator constantly changes depending on the steepness of the ground and also the performed operation. To maintain the machine’s balance, operators should ensure that the center of gravity does not go too far from the center of the machine, as the risk of tipping over is higher. We have created two concepts that indicate the balance of the excavator: ”center of gravity” and ”two tilting rates”. These names also imply what kind of information being visualized.

The first concept is called ”center of gravity”, since it shows the current position of the center of gravity with respect to the center of the machine from the top view. When the center of gravity is near the center of the machine (the circle in the center), it implies that the machine is in a very stable position (see the left image in Figure 7a). Each segment in this concept indicates the current position of the center of gravity. The left images in Figure 7b–d illustrate how this concept is visualized in some scenarios. Note that here we used the ground’s steepness as the influencing factor to simplify the illustrations. In practice, the operation being performed also influences the excavator’s balance.

The second concept is called ”two tilting rates”, since it shows the tilting rate of the machine from two different angles simultaneously: side view (top ones in two tilting rates) and rear view (bottom ones in two tilting rates). The reason behind this was the fact that excavators can be tilted forward-backward (side view) and left-right (rear view). The acceptable tilting rate varies based on the excavator manufacturer. Here, we used 25 ${}^{\circ }$ as the acceptable limit for illustration purposes. In this concept, we divided the tilting rate into five levels, which range from −25${}^{\circ }$ to 25${}^{\circ }$, and each segment represents one of these levels. One segment on both sides are illuminated together to visualize the tilt. For example, when the excavator is tilted to right side (see the right image in Figure 7b), the segments that indicate the rear view are illuminated to visualize the tilt, while the segments that indicate the side view remain stable. The opposite applies when the excavator is tilted backward (Figure 7c) or forward (Figure 7d), where the segments that indicate the side view are illuminated to visualize the tilt, while the segments that indicate the rear view remain stable.

3.3.3. Undercarriage Direction

This concept was made based on the top view of the excavator and each segment indicates the current direction of the excavator’s undercarriage when moving forward. In this concept, the cabin will always be facing forward, while the undercarriage can face to eight different directions. See the images in Figure 8 for some scenarios that illustrate how this concept works.

3.3.4. Relative Load Capacity

In addition to digging, excavators can also be used for lifting. When using excavators for lifting operations, it is important to know the permissible maximum load capacity that can be lifted. From the safety guidelines, we also learned that the maximum load capacity constantly changes depending on few factors, such as the height of the lifted object from the ground, the distance between the lifted object and the center of the machine, and whether the lifting direction is on the same way or across the undercarriage. The maximum load capacity is decreasing if the object is lifted higher, farther from the center of the excavator, or the lifting direction is not align with the direction of the undercarriage. The maximum limit for each influencing factor is usually recorded in a document called ”the load chart” provided by the excavator manufacturer. Operators are advised to refer to the load chart before performing lifting operations, since exceeding the limit will cause the machine to tip over.

The concept that we made is called ”relative load capacity”, which indicates how much load an excavator is lifting compared to the maximum load capacity. This concept consists of four different pieces of information: the angle of the boom, the height of the bucket/lifted object from the ground, the distance between the bucket and the center of the machine, as well as ten blocks that represent the relative load capacity, where each block indicates 10% of the maximum load capacity (see Figure 9). To avoid the machine from tipping over, operators have to ensure that the top block is not turned on. See the images in Figure 10 for some scenarios that illustrate how the relative load capacity works.

3.3.5. Generic Warning Sign

The last concept was a generic warning sign that only appears when a collision or loss of balance is imminent to occur (see Figure 11). When this warning appears, operators should stop their current action.

3.4. Interviewing Operators to Get Their Feedback

Since we did not involve any excavator operators in our design workshop, we interviewed seven excavator operators and presented the generated concepts from our design workshop to get their feedback. This activity was also done to answer “RQ4—How do excavator operators perceive the proposed visualization approach?”. All the interviews were done at the operators’ respective workplaces and each interview lasted for one hour. After collecting the informed consent from the operators, the interviews were started with collecting some background information about the operators (see

Table 1. The profiles of excavator operators that we interviewed.

No	Age	Status	Experience	Excavator Sizes	Knowledge on Head-Up Displays
1	22 years old	Professional	5 years	14–65 tonnes	Has tried one in a car
2	40 years old	Professional	2 years	1.5–75 tonnes	Has tried one in a car
3	43 years old	Professional	5 years	5–25 tonnes	Has no knowledge
4	53 years old	Hobbyist	11 years	1.1 tonnes	Has tried one in a car
5	58 years old	Professional	34 years	14 tonnes	Has seen one, but never tried
6	64 years old	Professional	50 years	1.5–95 tonnes	Has seen one, but never tried
7	39 years old	Professional	5 years	1.5–30 tonnes	Has tried one in a car

). We asked the operators about their age, experience, various excavators that they have operated (in terms of weight), and whether they were familiar with head-up displays. All the participants were males and six of them were working as professional operators. We also had one participant who owned an excavator and used it for personal use only, which we classified here as a hobbyist. One operator had no knowledge about head-up displays, while the remaining operators have seen advertisements or have tested one when test-driving a car.

We continued the interviews by explaining the meanings of the visualization concepts, which were printed on papers, to the operators. We also used some toys to help us in demonstrating the use of the visualization concepts, as shown in Figure 12. Once the operators have confirmed that they understood the logic behind each visualization concept, we then continued with some tests that evaluated the operators’ understanding. We showed some examples of the visualization concepts, which were also printed on papers, and then asked the operators to interact with the provided toys and move them around based on what kind of visualization that was shown to the operators (see Figure 13a–c). This approach was taken, since it would be easier for the operators to demonstrate their understanding through actions with the toys rather than to explain everything verbally. This approach is also different compared to the typical way of evaluating paper prototypes, where the researcher plays the ”computer” role and manipulates the paper prototype based on the input made by user [27]. As part of the evaluation, we showed ten examples using the proximity warning concept, eight examples for each concept of balance-related information, eight examples for the undercarriage direction concept, and ten examples using the relative load capacity concept. Only for the proximity warning concept, the provided examples had increasing difficulty, starting from one obstacle to multiple obstacles in different altitudes. The remaining visualization concepts did not have increasing difficulty, since the complexity among the different examples was mostly the same. Note that the generic warning concept was not evaluated, since its meaning was too obvious for the operators. Some of the provided examples are already shown in Section 3.3.1, Section 3.3.2, Section 3.3.3 and Section 3.3.4.

In the last part of the interviews, the operators were asked to choose which visualization concepts that they would like to have and where the concepts should be placed on the windshield. The visualization concepts were printed on a transparent film and cut into several pieces, which could be move around. We then presented a paper that has an image of excavator’s cabin printed on it. When evaluating a paper prototype, it is also common to provide some overlaid materials that the user can interact with, since this approach could provide the user some ideas on how the final interface may look like [28]. We then asked the operators to place the visualization concepts on the windshield according to their preferences. The operators were also free to exclude some of the visualization concepts, if they thought that some of the visualization concepts were unnecessary. Finally, the operators were asked to describe the reason behind the placement that they have made.

4. Results

This section presents the feedback on the proposed visualization concepts that we received from the operators, how they were revised, and where they should be placed on the windshield.

4.1. Comments on the Proximity Warning

The operators could understand this concept without any problem when the obstacles existed on the left side or the right side only (see Figure 6a,b for the examples). The operators started to misinterpret the concept when there was an obstacle exactly in front or behind the machine. Although it was previously explained that, as the proximity warning was split into left and right sides, the same segment on both sides would be lit when the obstacle was exactly in front of or behind the machine (see Figure 6c for an example). The operators said that it would make more sense if each segment represents the obstacle that exists in one direction (see the images in Figure 14 for the illustration of this phenomenon). We then realized that this way of thinking was not wrong either, since if there are two obstacles with similar proximity, where one is on the left side and another one is on the right side, then the visualization will look like the right image in Figure 14. The operators further suggested that they would prefer to have the proximity warning concept in a complete circle, instead of separated half circles.

The operators could also use this concept well when the obstacles were on the same level. When the obstacles were on different altitudes (see Figure 6d for an example), they could pinpoint both position and proximity of the obstacle, but failed to determine whether the obstacle was lower, on the same level, or higher than the machine. They said that it was too difficult to process the position and the altitude separately. Operator #3 even suggested that it would be useful if both position and altitude of an obstacle can be somehow indicated using the same segment. Despite the importance of avoiding any collision with nearby objects, Operator #5 commented that he did not want to have the proximity warning concept presented on the windshield, due to its current complexity. The remaining operators still would like to have this concept presented on the windshield, given that the design could be revised in order to reduce the complexity.

We then incorporated the feedback from the operators by revising this concept into a complete circle with segment groups that point to eight different directions (see Figure 15a), thus the ambiguity problem shown in Figure 14, when an obstacle is in the front of or behind the machine, is solved. Unfortunately, we could not incorporate the comment from Operator #3 about indicating both position and altitude of an obstacle using the same segment, since it would make the design even more complex. To reduce the complexity of the design, we decided to remove the segments that indicate the altitude of the obstacle the obstacle. This decision reduced the number of information that the operators should acquire and process.

4.2. Comments on the Balance-Related Information

All the operators could use and understand both concepts without any problem. However, the concept of center of gravity was preferred by six operators, while only one operator who preferred the concept of two tilting rates. Regarding the concept of two tilting rates, six operators said that it was slightly more difficult to process two pieces of information simultaneously. Regarding the preference towards using the concept of center of gravity, Operator #7 even commented that operators only need to know where the center of gravity currently is and they do not need to know how much the machine is tilted. Operator #2 further commented that, for both concepts, the segment that indicates that the machine is in a stable position could be removed (see Figure 7a for an example), since that means there is nothing that operators should concern about. Since the operators preferred the concept of center of gravity and the feedback from Operator #2 did make sense, we revised this concept by removing the segment that indicates the machine is in a stable position (see Figure 16). This means that one of the segments will be turned only when the machine started to tilt.

Regarding whether either of the concepts should be presented on the windshield, Operator #1 decided to exclude any balance-related information. He argued that an experienced operator could feel the balance of the machine through the movement of the cabin, and thus such information is not needed for him. The other operators who included one of the balance-related information also commented a similar thing, but they argued that it is nonetheless nice to have something inside the cabin that could validate their gut feeling.

4.3. Comments on the Undercarriage Direction

This concept was understood instantly by the operators, and thus there was no further feedback regarding how the information should be visualized. However, only three operators who would like to have this concept presented on the windshield, while the remaining operators considered this information unnecessary, especially for experienced operators. Among three operators who included this concept, Operator #2 and Operator #4 said that it is a nice piece of information to have, although not very critical. Operator #7 also argued that this information would be useful when operating a large excavator, since operators can see the undercarriage, but the direction, where the undercarriage will move, is not always clearly visible.

4.4. Comments on the Relative Load Capacity

All the operators agreed with the logic behind this concept and all of them would like to have it presented on the windshield. However, the operators commented that there is too much information in this concept. The operators explained that the most important part in this concept is the ten blocks that represent the relative load capacity, as these blocks are the information that indicates whether the machine will collapse or not. The operators had different opinions whether to keep or remove the information about the height of the bucket/lifted object and the distance between the lifted object and the center of the machine. Although not always needed, both pieces of information would be useful when precision is needed. However, all the operators agreed that the information about the angle of the boom is unnecessary, and thus it could be removed. Considering the unanimity among the operators, we revised this concept by removing the information about the angle of the boom, while the other kinds of information remain the same (see Figure 17).

4.5. Comments on the Generic Warning Sign

Even though this concept was not evaluated, all the operators understood its meaning instantly by looking at it. Six operators would like to have this information on the windshield, while Operator #3 would like to have this concept presented on the head-down display that exists in the cabin, as the head-down display can show something flashy that could attract his attention. Operator #3 further suggested that it would be good to have two versions of the warning sign: one is for the imminent collision and another one is for the loss of balance. Based on this comment, we decided to have two versions of the warning sign, where one is specifically for the imminent collision and the other one is for the loss of balance (see Figure 18). Therefore, it is much clearer for what kind of incidents that the warning represents.

4.6. Information Placement on the Windshield

As the last part of the interviews, the operators were asked to choose which visualization concepts that they would like to be presented on the windshield and where they should be placed. Based on Figure 19, we could see that the operators were quite positive with this approach, since all of them decided to include at least three visualization concepts to be presented on the windshield. According to the placements that the operators have made, we could see that there is a pattern, where most of the information should be presented peripherally (see the images in Figure 19). The operators commented that they need to have the working area, for example, the area where the bucket is, to be free from any obstruction. However, there is an exception to this, as five operators would like to have the generic warning sign to be presented centrally, as shown in Figure 19. The operators argued that, as they pay attention to that area the most, placing the warning sign there will allow them to detect the warning instantly. Regarding the placement of other visualization concepts, we unfortunately could not get a firm indication from this study, as the operators’ preferences were quite diverse.

The operators also commented that the placement of information highly depends on the operation being performed. For example, when digging, the operators would spend most of their time looking down. In this case, it would be appropriate to have the information in lower positions. Similarly, the operators would mostly look above in a lifting operation. In this case, it would be useful to have the information in higher positions. The operators further commented that it would be great if the information can be moved around the windshield depending on their needs. Although this suggestion is reasonable, transparent displays are unfortunately designed to be installed in one fixed place, which is similar to how head-down displays and projection head-up displays are installed inside the cabin. Changing the placement of transparent displays is technically possible, but operators would need the help from a technician to do so.

5. Discussion

In this study, we used four safety guidelines to find what kinds of information that excavator operators should know to perform their work safely. Using this approach, we were able to come up with the list of information that could be relevant for operators, regardless of their operational preferences. In the end of the interviews, we also asked the operators if there were other kinds of information that they would like to have, but were not presented in this study. All the operators said that the list of information that we presented was sufficient.

Although the proposed visualization concepts were presented in the form of papers, all the operators showed engagement and we also managed to get valuable feedback on the proposed visualization concepts, as presented in Section 4.1, Section 4.2, Section 4.3, Section 4.4 and Section 4.5. As the operators were provided with some toys to interact with, it was easier for them to convey their understanding on the proposed concepts. The operators could just move the toys around to show what they meant, instead of describing everything verbally. It also worked the other way around, since we could easily confirm the operators’ answers through the toys, and thus preventing any misinterpretation to occur.

We used a paper-based prototype to propose and evaluate the visualization concepts, since we wanted to get feedback from the operators, where modifications could still be easily incorporated. Our approach was in contrast with the common practice in the heavy machinery domain, since the operators were involved after everything has been developed, as discussed in Section 2. Through this study, we at least managed to confirm that there is a match between the presented information, the performed operation, and the operators’ way of thinking. As presented in Section 4, four out of six visualization concepts generated in this study could be understood easily by the operators. Nevertheless, the reality is more complex and the actual excavator operation would also influence how operators acquire and utilize the presented information. Based on our prior research [4], we found that the information acquisition rate was decreasing when the situation complexity was increasing. However, even in such condition, the information acquisition rate for the head-up display was still higher compared to the head-down display.

The designs of the visualization concepts in this study were made based on a commercially available technology. This choice was made after careful consideration on both advantages and disadvantages of every commercially available technology, as discussed in Section 3.2. Furthermore, we also wanted to propose something that may actually be deployed not only in simulated environments, but also in real excavators in later stages, similar to what has been done by Rakauskas et al. [17] in snowplow trucks, Englund et al. [18] in forest harvesters, and Fang et al. [14] in mobile cranes. In the context of excavators, prior studies were still conducted in the simulated environments. Akyeampong et al. [13] visualized the information directly within the simulated environment, and thus it is unclear how the visualization would work in practice and what kind of technology would be needed to realize it. In our prior work [4], we used a projector to present the information on the windshield and we were aware that this approach could have disadvantages in bright environments.

Considering the feedback from the operators, as described in Section 3.3.2, that they could know the state of the machine based on the machine’s movement, it indicates that other modalities, such as haptic or auditory, could also be used as an alternative or complement to the visual information. However, it is important to note that excavators already produce loud noise and strong vibration due to the working engine and the performed operation [29,30]. Therefore, increasing the number of haptic or auditory information should be done cautiously, since multiple modalities may cause conflict and interference, which could deteriorate operators’ performance [31]. While the mainstream approach for presenting information in heavy machinery is still mainly using visual information [8], there were studies that investigated the use of force feedback or auditory cues to complement existing visual information in excavators. Regarding the use of force feedback, there are challenges on determining the appropriate way of delivering force feedback [32] and improving the ergonomics of haptic devices [33]. Desai et al. [34] investigated the use of auditory cues in excavators and the result indicated that operators performed better when using the auditory cues. However, as the study was done in a simulated environment, it is unclear to what extent the auditory cues would work in real operations.

6. Future Work

As this study was still limited to the match between the presented information and the operators’ contextual needs, high-fidelity prototypes are nonetheless needed to further evaluate the proposed visualization concepts in some operation scenarios. We would then then be able to determine the impact of having such visualization on operators’ performance in certain scenarios. One example could be the difference on the information acquisition between situations with less things to be observed, such as operations in mining sites, and situations with more things to be observed, for instance, operations in urban areas. Since we proposed several visualization concepts in this study, it would be interesting to firstly evaluate them one by one to determine how each visualization concept would help operators to perform their work. After that, we could try presenting multiple concepts simultaneously to investigate if there would be conflict or interference due to the presence of multiple visual information. Depending on the fidelity of the prototype, future evaluations could be done in simulated environments or real-world settings. Through this study, we were unable to determine the optimum information placement on the windshield. Taking that into account, future evaluations could also be carried out to determine which placement of information that will provide the optimum result on operators’ performance, similar to what Topliss et al. [35] has done in the automotive context.