The research questions Q1, Q2, and Q3 were addressed in order to identify the opportunities, benefits, and constraints of using virtual reality for usability testing and design reviews in industry. In the sections that follow, we analyze our findings.
3.1. Patents Landscape
The seven patent records retrieved by the search strategy are shown in
Table 1.
To address the first research question, these patents were analyzed to answer frequent concerns and uncover patterns in assignees, filings per year, the International Patent Classification (IPC), and benefits.
Q1: How are patents in industrial virtual-reality-based usability testing and design review characterized?
In terms of assignees, their identification may help the discovery of industry leaders, the evaluation of possible rivals, and the identification of niche players.
The widely fragmented dispersion of patent applications across many assignees is intriguing: just one assignee, CCB Fintech and China Construction Bank, submitted two patents, whilst the other patents were filed by different assignees.
Rather than a relatively equal-sized, but large, portfolio held by a few companies, which indicates an active competitive market with strong investments by multiple companies, suggesting that the market is difficult to enter, we discovered a large number of assignees, each with a small number of records, demonstrating a developing technology space. This domain may be entered by acquisition or rapid development. There are several companies, each with a small number of patents, indicating an opportunity to enter this area while it is still young, either by licensing existing technology, purchasing one of the players, or developing new technology that is not already patented.
It is also worth noting that three of the six assignees are major financial institutions, while one university filed one of the seven patents.
In terms of patent filings per year,
Figure 2 illustrates the yearly filing of patents from 2016 to 2020. We omitted 2021 patents from the analysis since they were still being filed at the time of this study.
We observed an initial period spanning the years 2016 and 2017, with no patents registered. This trend changed in 2018, followed by growth in 2019 and 2020.
These results support the claim in [
28] that the year 2016 marked a technological breakthrough in the domain of virtual reality. Before 2016, commercial virtual reality systems required users to connect a headset, controllers, and sensors to an external high-end computer, which was an expensive, bulky, and inconvenient setup. Thus, the current all-in-one virtual reality systems are a significant step forward from only a few years ago [
38]. The upward trend slowed in 2020, although one likely explanation is the 18 month patent legal secrecy restriction.
A important trend is that not only is the global virtual reality market projected to grow from
$6.30 billion in 2021 to
$84.09 billion in 2028 [
18], but this rise may be much greater, given that the COVID-19 epidemic boosted the usage of virtual reality even further [
39]. As a consequence, this scenario suggests that the exponential increase in patenting filings related to virtual reality and design review and usability testing will continue for a few more years.
The International Patent Classification (IPC) is an approach to determining a standard classification for registered patents, thereby enabling the search for and access to technical information accessible in documents connected to the same subject. The system is a hierarchical patent classification system used in over 100 countries to uniformly classify patent material. It creates a separation into classes and subclasses that are applicable to various technical domains and aids in the standardization of patent classification.
Figure 3 illustrates the IPC classifications that we found in our review.
Section G (Physics), which covers all physics-related material, is designated to all seven patent filings. As the class level progresses and technological information is described in more depth, the most frequent class is G06 (computing calculating or counting), with subclasses G06F (electric digital data processing), G06Q (data processing systems or methods), and G06T (image data processing or generation, in general).
Other patents grouped under G Section are G02B (optical elements, systems, or apparatuses), G09G (arrangements or circuits for control of indicating devices using static means to present variable information), and G10L (speech analysis or synthesis; speech recognition; speech or voice processing; speech or audio coding or decoding).
Section A (Human Necessities) was also used to classify one patent, grouped under the A61 class (medical or veterinary science; hygiene), with subclasses A61B (diagnosis; surgery; identification) and A61N (electrotherapy; magnetotherapy; radiation therapy; ultrasound therapy).
In terms of benefits, we examined the seven patents in relation to the advantages of each invention, as described by its authors, and the novelty of each invention, i.e., the unique innovative feature introduced by the inventor that is not conventional and constitutes an improvement on existing technology.
The JP2021068278A patent [
33] proposes to provide a design review system and method that reduce the time required for computer-aided design (CAD) data conversion. The proposed design review system includes a CAD apparatus, into which CAD data are produced or edited, and a data conversion unit that performs the data conversion from CAD data to virtual reality data. The advantage of the invention is that the design review system shortens the time required for data conversion, such as data conversion from CAD data to virtual reality data.
The US20210011593A1 patent [
34] proposes a system for producing applications based on real-time accessibility assessments. The system identifies that a user is accessing an application on a device (such as a virtual-reality device), captures the real-time accessibility data, inputs it into a machine learning model, generates an accessibility score, and renders the application based on this score. The advantage of the invention is that the system identifies whether the user is accessing another application on the same user device and utilizes the accessibility score stored in a data repository to render the other application; if the amount of data stored in the data repository increase over a period of time, this improves the efficiency with which they are processed. The method enables the calculation of a real-time accessibility score to comprise the operating configurations of the user device, as well as the status of its hardware components, to establish internet connectivity in an effective manner.
The CN111414084A [
32] and CN111414083A patents [
31] are related. The former proposes a space usability testing laboratory comprising a test area for displaying a 3D model of the space based on immersive virtual reality, and the experience data of test users are collected during the test process [
32]. The latter proposes a method for the usability testing of the space, which involves obtaining a 3D model of the space to be tested, and the test task is performed by a user wearing a virtual-reality-based wearable device [
31]. According to the inventors, the advantages are that the user’s understanding of the user experience may improve this experience, space usability testing is more convenient, and its results are more accurate. Furthermore, the manner of implementation could truly display the design appearance, the user experience is real, the test result is accurate, and the test consistency and efficiency are high.
The US20210090343A1 patent [
35] proposes a method for providing design reviews using virtual-reality devices, involving the processing of interactions of users in a virtual-reality format and the generation of output on the basis of users’ actions. The advantage of the invention is that the method would make it possible to generate a complete design review cycle in an easy manner, so that time consumption could be reduced effectively. The inventors claim that the technology would open the door to a range of new applications that have not been possible until now and that the invention would revolutionize design reviews with a radically new experience using virtual reality. Multiple users might collaborate remotely and perform design reviews within the virtual reality environments.
The US20190227626A1 patent [
36] proposes a system for personalizing a human–machine interface (HMI) device based on the mental and physical state of a user. During the performance of a task in a simulation environment (such as training in virtual reality), the system extracts biometric features from data collected from body sensors and brain entropy features from electroencephalogram signals. Both data are correlated to generate a mental-state model. The mental-state model is deployed in a, HMI device during the performance of a task in an operational environment for the continuous adaptation of the HMI device to its user’s mental and physical states. The advantage of the invention would be that the continuous adaptation of the HMI to the mental and physical states of the user would reduce the workload and enhance decision-making. The need for unnecessary modifications in the interfaces would be eliminated, and the designs would be more user-centered and customized to the real needs of users.
Finally, the KR20190088710A patent [
37] proposes a method for assessing the usability of automotive infotainment systems that comprises facilitating interactions between numerous virtual infotainment systems and drivers in a virtual driving environment while investigating the usability of vehicle infotainment systems. The usefulness of a virtual infotainment system is determined by evaluating both the execution time of the operation command and the running condition of a virtual car as a result of the execution of the operation command. The virtual car’s running state contains the virtual vehicle’s speed, its distance from the preceding vehicle, the distance between the lane and the virtual vehicle, and the steering angle during the execution of a drive operating instruction. The novelty of the invention is its evaluation of the cognitive load of a driver by using the vehicle infotainment system and its use of the driving state of a virtual vehicle in accordance with the performance time of the operation command and the performance of the operation command to evaluate usability for each of virtual infotainment system.
3.2. Scientific Mapping
Table 2 shows the twenty articles selected by the search strategy.
These articles were reviewed in order to answer the research questions Q2 and Q3. It is worth noting that the attributes addressed by Q2 and Q3 are not abstracted at the same level and are not always mutually exclusive in the studies reviewed; furthermore, some studies did not mention some of these aspects. Our findings are provided in the subsections below.
Q2: In terms of the application fields, methods, hardware, and software involved, how is current knowledge on the application of virtual reality in usability testing and design review in industry defined?
3.2.1. Application Fields and Methods
The application field of an article was defined similarly to [
56]’s approach as the industry and/or technical environment targeted by the study. One of the twenty articles selected is a literature review [
49] that assessed some of the studies under investigation. As a result, we omitted it from our quantitative analysis of application fields in order to avoid counting the same study twice.
The application fields observed can be divided into three main categories: (1) automotive industry, (2) industrial machinery manufacturing, and (3) laboratory environment or field undetermined.
Figure 4 shows the distribution of papers in each category.
The automotive industry was the subject of seven articles (37%) [
43,
46,
48,
50,
52,
54,
55]. It was expected that a substantial number of virtual reality studies would be conducted in the vehicle industry. These findings are congruent with those of [
56]. The importance of the automobile industry may be related to the fact that virtual technologies have long been utilized in this sector in a range of specialties and applications, such as manufacturing, training, and maintenance, to mention a few [
13,
57,
58]. Departments such as design, engineering, maintenance, and assembly are already using the technology to support practically everything in the automotive sector, from product development to task assistance in machine assembly or maintenance procedures. Furthermore, the automotive industry is one of the most mature manufacturing industries, with cutting-edge technologies being used for the first time on a regular basis [
56].
Regarding the evaluated systems, three studies examined the virtual interaction with an automobile multimedia system, utilizing distinct basic functionalities and, as a result, producing diverse usability testing scenarios. Among the functions are a navigation system, air conditioning, a phone, a radio, a driving assistance, and car parking. Other systems undergoing design review or usability testing include a BMW vehicle’s exterior design and Audi [
46] and Volvo multimedia stations.
Some usability tests guided interaction operations in which participants were immersed in dynamic virtual environments, such as driving a vehicle in a virtual city [
46]. For dynamic testing [
46,
52], participants drove on public roads in a real-world setting, interacting with the vehicle’s panel controls and performing tasks as assigned by the moderator. Each participant interacted with the digital system while driving around virtual streets in the virtual usability testing environment, while a moderator observed the participant’s behavior and collected metrics.
Two studies did not consider how individuals interacted with a product prototype. The authors of [
43] focused on design issues, asking their participants to utilize a variety of assessment techniques to determine the best way to interact with a virtual automobile model. The following exchange possibilities were investigated in the study: voice command, gestures, first-person vision, and physical controllers.
Due to the variety of systems studied, several operations, such as air conditioning, navigation, telephone, and audio controls, were used by various studies in diverse contexts of use, command design, and visual representation. This limits the evaluation of the connection between the results obtained in the research [
46,
48,
52], since the use of standard methods for usability testing would benefit from a thorough examination of the data and conclusions gathered from various investigations.
Five studies (26%) reviewed industrial machinery manufacture. Three of them [
15,
16,
45] were published by the same research group and focused on industrial power units, while the other two looked at hydraulic pump production [
41] and specific machines for automation technologies [
53], respectively.
Seven (37%) of the studies did not indicate the field of application [
19,
20,
40,
42,
44,
47,
51]. It is common for virtual-reality applications developed by an academic research team to be tested in their own laboratory with prototypes and artifacts [
59].
Figure 5 is an example of a study conducted in a laboratory environment.
In terms of the methods and metrics used, we identified that the studies focused on the accuracy of experimental results when compared to testing with conventional CAD models and physical prototypes. The articles under consideration spanned a broad range of methods and metrics. The main approaches used by the studies under examination are listed in
Table 3.
Despite the fact that the reviewed studies utilized a wide variety of methods for collecting qualitative and quantitative data, some of them were referenced in several studies.
Three studies used the Time to Complete the Task metric. The Time to Complete statistic is used to calculate how much time each participant spent on each task.
The qualitative data were often gathered by providing a standardized questionnaire to respondents after they completed the tests. The System Usability Scale questionnaire was often used to measure the perceived user-friendliness of the system with which the participant interacted. The SUS approach is frequently used in industry and is effective for system comparison [
46].
Another often-used metric is the Number of Mistakes, in which moderators utilize visual observation to quantify the mistakes made by participants while performing tasks other than those for which they were trained.
The Rapid Upper Limb Assessment score was used in two studies. The RULA is one of the most frequently used measures for assessing employees’ exposure to ergonomic risk while performing manual upper body activities such hand, neck, and limb twisting [
46].
The Sense of Presence Inventory (ITC) and User Experience Questionnaire (UEQ) are two methodologies worthy of note. The Sense of Presence Inventory is a metric used to examine how people perceive physical space (defined as “a sense of physical placement in the mediated environment and interaction with and control over parts of the mediated environment”) [
52]. The User Experience Questionnaire is used to obtain evaluations from participants on the following aspects: attractiveness (overall impression of the product), perspicuity (ease with which the product can be understood and used), efficiency (ability to use the product efficiently), dependability (the feeling of being in control of interactions), stimulation (excitement and motivation to use the product), and novelty (perception of the innovation and creativity of the product).
Other methods and metrics were only referenced in one study, such as FMEA (failure modes and effects analysis), criticality analysis (CA), and expected final distance [
53]; intuitiveness and task weight [
45]; readability of information, command display, and function controls [
44], as well as several others, such as heat maps, pupil diameter, heart rate (hr), breathing rate (br), activity (vmu), posture, heart beats per minute, breath per minute, etc.
Table A1 lists the methods and metrics specified in each of the articles reviewed.
Some studies had methodological limitations that may have influenced their findings, specifically the relationship between the usability testing results in physical prototypes and virtual prototypes. In certain cases, participants had prior interactions with the systems being assessed, which influenced the evaluation of the system’s ease of use in the virtual environment [
45].
Other methodological limitations that we observed relate to the quantity and profile of the participants in usability testing.
The authors of [
60] caution against involving people who have a link to the product under development, such as corporate executives. Employees’ involvement in usability testing may be unconsciously impacted due to their relationship to the business, affecting the dependability of the data acquired. As a consequence, the authors recommend that the external target group, or future customer, participate in usability testing at some point throughout the development process. However, regarding the respondents’ profiles, the usability testing included personnel from Volvo [
52], BMW [
43], and Audi [
46], as well as university students [
48].
Additionally, the quantity and profile of participants in usability testing may affect the accuracy of collecting specific metrics, thereby reducing the repeatability of the usability testing findings when repeated with different participants. For the SUS metric, for example, thirty-five individuals are recommended for experiments in order to obtain satisfactory results [
61].
Table 4 shows the number of participants in the examined studies.
3.2.2. Hardware
Because virtual reality is a complex technology that combines interactive media, sensors, displays, human–machine interactions, simulations, computer graphics, and artificial intelligence technologies to expand human perception, virtual reality systems frequently require the use of multiple devices [
56]. Depending on the application, virtual reality hardware may range from a basic computer to specific display devices, motion capture equipment, and interactive gadgets, such as wearable devices, cameras, head-mounted displays (HMD), and so on.
Table 5 lists the hardware, gadgets, and apparatuses referenced in the evaluated publications.
We found a vast variety of technologies, both in terms of the devices and the models employed in each category. Some studies did not specify which model was used. We discovered that the most frequently utilized hardware may be classified into seven basic categories: (1) head-mounted displays (HMD), (2) motion capture systems, (3) cockpits, (4) sensors, (5) automatic virtual environments (CAVEs), (6) interaction devices, and (7) glasses. These categories are not mutually exclusive, and the arrangement is often built by combining various technologies.
Both visualization and tracking technologies are required in a virtual-reality environment. Head-mounted displays and projection-based systems are the most frequently utilized virtual reality visualization technologies in the industrial sector. Head-mounted displays are devices that are affixed to the head of a virtual reality user and generally feature one or two screens as the image source, as well as a collimating lens between the eyes and the display [
62]. Projection virtual-reality systems, on the other hand, include single or multiple projector-based powerwalls, as well as surrounding, walk-in installations based on numerous projection screens (e.g., CAVEs).
The authors of [
56] noted that the usage of CAVEs and head-mounted displays is unusual in industry, which they attribute to high costs. However, since head-mounted displays were the most widely used devices in the research we considered, we discovered a distinct situation. One likely reason is that, until recently, the use of head-mounted displays was restricted due to their high cost and technological restrictions.
However, there was a technical breakthrough in 2016, with the first public release of technologically mature virtual-reality head-mounted displays, such as the HTC Vive and the Oculus Rift [
27,
53]; since then, there has been an increase in worldwide research on virtual [
28]. Not only academics, but also practitioners agree that the equipment launched in 2016 was a “very big breakthrough” for virtual reality applications [
29,
30]. Therefore, although the cost of these devices remains relatively expensive and technical limits exist, the price of virtual reality equipment has reduced year on year, technological constraints have decreased and new features have been developed, resulting in the increased use of head-mounted displays.
The HTC VIVE was the most commonly reported model in the studies that employed head-mounted displays. Ten of the studies examined used HTC VIVE, well ahead of the second most frequently mentioned model, the Oculus Rift.
The authors of [
15,
16,
45] stated that they chose HTC Vive because its tracking sensors work reliably and its controllers allow multimodal hand inputs. Furthermore, HTC Vive supports development in the Unity3D game engine, which has become the standard for the development of virtual reality.
The HTC VIVE is a virtual reality headset that consists of a head-mounted display, two wireless handheld controllers, and two lighthouse base stations that emit pulsed infrared lasers. It allows the user to move about and interact with a 3D world using motion-tracked handheld controllers. The VIVE system has two 1080 × 1200 resolution displays, one for each eye. The headset and controls also have 70 infrared sensors, a gyroscope, and an accelerometer. These sensors, together with the two lighthouses, track the operator’s motions with millimetric precision. The operating system is SteamVR, which runs on Microsoft Windows. A USB connection attaches the VIVE system to the computer [
63].
Figure 6 is an example of a study in which HTC VIVE head-mounted displays were employed.
Three of the studies examined employed immersive rooms and CAVE-like systems. It is worth noting that all three were published in 2017, shortly after the previously mentioned launch of accessible head-mounted display releases in 2016. Prior to the popularity of head-mounted displays, CAVE equipment was the most frequently utilized technology [
18]. However, since CAVEs are expensive, have poor immersion, and are not especially portable, they have been increasingly replaced with head-mounted displays. These claims are supported by the authors of [
64], who found that in 2022, 28% of industrial presentations used a virtual reality system, whereas just 4% used a CAVE setting.
Motion capture devices are also frequently mentioned. Because sensing in single postures while handling a virtual object is currently inadequate, motion capture devices are utilized to supplement the detection of virtual world players and to reduce occlusion problems. Microsoft Kinect and Vicon models were utilized, with Kinect being used in two different studies by the same research group.
Cockpits are regularly cited devices in research on automotive applications. During testing, physical structures equipped with steering wheels, benches, and pedals were mixed with virtual reality elements. A cockpit with a head-mounted display device and sensors to detect hand motions was a common combination. In some experiments, vehicles with their original multimedia systems were employed in comparative evaluations of usability testing in real environments. Participants traveled along simulated routes while engaging with the vehicle’s interior controls. The use of physical device, such as seats, steering wheels, pedals, and gear shifters, allows the user to interact with the virtual environment in a similar way to haptic devices, which are capable of increasing the participant’s sense of immersion during the test and, in some ways, supporting the tactile feedback of the virtual-reality experience.
We observed the use of other hardware with specialized functions, such as Nintendo Wii remotes, Xbox controllers, and Leap Motion to improve interactions.
A wide range of other gadgets, including 3D printed rigid bodies with markers [
54], GoPro cameras [
54], and 3D laser scanners [
55], were only mentioned in one study.
Table A1 lists the hardware specified in each of the articles reviewed.
3.2.3. Software
Similarly to what we identified on hardware devices, we discovered the use of a broad variety of editors, programs, engines, and frameworks. The software adopted, like the hardware, is not mutually exclusive; rather, we found that there is typically a combination of various complementary solutions. The software referenced in the articles examined is shown in
Table 6.
Game engines are the most often used virtual reality software. The authors of [
54] also found that gaming engines have been used in virtual manufacturing in recent years. These game engines may provide useful application programming interfaces for rendering, interaction, and physics, allowing for the creation of high-quality and high-fidelity virtual environments, as well as extensive online documentation [
56,
65].
Unity 3D, a game production platform that incorporates a game engine, was mentioned by ten studies and was the most widely used product. The authors of [
15,
16,
45] declared that they chose Unity3D because it allows the integration of scripted behavior and offers a simple import workflow for 3D data.
Only one study used the game engine 3DVIA Virtools rather than Unity 3D. The authors explained that 3DVIA Virtools was chosen to develop the virtual-reality application because it can be built on top of a sophisticated BMW shader library and production data import workflow, and it has a flexible graphical scripting language that allows the programming and refining of interaction logic at run-time.
In terms of CAD and modeling tools, three studies produced by the same research group used CATIA [
15,
16,
45], and Siemens JACK was used in the experiment in [
54] for product digitization.
A wide variety of software, such as plugins, editors, and libraries, among many others, were cited by only one study.
Table A1 lists the software specified in each of the articles reviewed.
Q3: What are the benefits and challenges of using virtual reality for usability testing and design review in industry?
The examined articles highlight different perspectives in which virtual reality might overcome the constraints of conventional design reviews and usability testing processes.
According to the findings, virtual reality has two key advantages: benefits over conventional CAD models on screens and benefits over physical prototypes. In terms of advantages over CAD on screens, the advantages highlighted include the visualization of data from different angles and on a true scale, increased team collaboration and feelings of engagement, and more intuitive and natural interactions for non-CAD specialists. The benefits over physical prototypes include cost and time savings on redesign and increased safety. These advantages are discussed in depth below.
3.2.4. Visualizing Products from Different Viewpoints and on a True Scale Stimulated Novel Insights
The most frequently cited benefit of virtual-reality reviews of CAD on a screen is enhanced 3D visualization and manipulation. The conventional review process is often carried out on a computer with the use of CAD tools, on a flat screen. The visualization of CAD on a screen may not always satisfy all of the criteria for functional and ergonomic validations of complicated 3D models [
16]. Virtual reality allows more novel modes of visualization and interaction to enhance engineering design reviews in this situation [
16].
Several studies reported that the opportunity to observe a product from different angles and with more detail generated innovative discoveries [
55]. The authors of [
41] observed that visualizing planes in a virtual reality environment gave participants a better understanding of the spatial relationships between product components, as well as the interaction space around the assembly line, allowing the design team to understand the operating clearances in real size. The authors [
42] found that relevant information about transport routes or kinematic properties, which is either not modeled in the CAD data or is lost during conversion processes, played an important role in reviews and provided an observation of machine mechanics (e.g., motor-driven mechanisms). The authors of [
41] observed that the possibility of viewing and interacting with the geometry of the product (a pump) and the surroundings (an assembly line) on a true scale benefited the team in understanding important viewability problems during a subassembly engagement. The team received kinesthetic and ergonomic information on operator movement via natural engagement with the Wii remote [
41].
On average, we found a consensus that this new visualization facilitated overall mistake detection. The authors of [
16] found that, when compared to a standard CAD software approach on a flat computer screen, participants are more likely to spot faults in a 3D model inside an immersive virtual reality environment. In [
41], the team uncovered design flaws and possible solutions that could not be detected or verified using conventional computer tools. Virtual-reality-enabled design review enables users to detect significantly more flaws in a 3D model than a CAD-software-based approach on a PC screen [
16].
The benefits of combining fully digital CAD models with physical components of hybrid prototypes, such as the cockpit gear utilized in the automotive industry, are also emphasized. With the addition of the physical model, designers can evaluate their designs both visually and tactilely. This adds a further physical dimension, allowing designers to not only “see” their designs, but also “touch” them, providing designers with simulated interaction solutions in the early stages of design [
48]. In [
19], it is argued that virtual reality’s benefits contribute to the Industry 4.0 and cyber–physical system issue of linking the physical and digital worlds.
3.2.5. Increased Team Collaboration and Feeling of Engagement
The advantages of improved collaboration and engagement were also cited frequently in the research reviewed.
Industrial design reviews and usability testing are complex processes involving a variety of stakeholders, including designers, engineers, and end users. Computer-aided design is utilized as a communication tool in a conventional review process to transmit design ideas and enable a better common understanding among diverging needs and perspectives [
15]. Because virtual reality decreases the possibility of some groups being excluded from the review process, it has the potential to foster collaboration among stakeholders [
16].
According to some studies, the high focus that virtual-reality-based reviews provide [
52] increases a feeling of team engagement. In [
41], it is asserted that in the conventional process, design teams cluster around conference tables with laptops, mobile phones, and paper notes while, at best, one person manipulates the design on a giant 2D screen. Maintaining team engagement and attention becomes more challenging when battling with technological device distractions. The virtual environment allowed the design team to move away from the traditional conference room and into a creative area with fewer distractions. During design discussions in the immersive environment, the team observed increased engagement. Thus, the immersive virtual reality environment enhanced team engagement, which resulted in better discussions and fuller participation from team members in decision-making [
41].
Figure 7 shows a design review on a screen and a VR-based process.
3.2.6. More Intuitive and Natural Interactions for Non-CAD Specialists
According to many studies, virtual-reality-based design evaluations and usability tests are more friendly and intuitive. While CAD software does not allow the intuitive analysis and manipulation of 3D models by users without a CAD or computer science background [
15], interaction in virtual reality environments is generally simple and intuitive [
16,
19], and 3D engineering data can be visualized in virtual reality. Because of the high level of immersion provided by virtual-reality head-mounted displays, conducting design reviews and interacting with 3D models is regarded as more intuitive and “natural” for non-CAD specialists [
16].
The authors of [
16] observed that the intuitiveness of interactions in a virtual-reality system enabled a considerably faster entry into the design review.
3.2.7. Cost and Time Savings for Redesign
When compared to the conventional review process of physical prototypes, the most frequently noted advantages of virtual reality are its cost and time savings. According to the research analyzed, industries may employ virtual prototypes to save money [
43,
46,
48], minimize redesign time, and expedite time-to-market.
Prototyping is an essential step in the product development process. However, after building a product model, testing its design and functionality requires time and money. In this context, a virtual prototyping system based on virtual reality technology has the potential to overcome these shortcomings [
48].
The authors of [
54] also describe the benefits of lowering the time for product design reviews and the amount of design and engineering design changes, as well as reducing time-to-market and optimizing costs.
Immersion in virtual reality may result in cost savings as well as better and/or quicker design review processes. Because physical prototypes and mock-ups may be replaced by their virtual counterparts, virtual reality improves design verifications and the review process, which might contribute to cost savings for manufacturers [
13,
19].
In [
46], it is claimed that the principle of “simultaneous engineering”, in which elements are designed and tested virtually concurrently with vehicle development, is achievable with, and potentially strengthened by, the use of virtual reality [
46]. The authors use the automotive industry as an example to demonstrate that a type of control could be created early and parallel to the development of a new car model and tested within a virtual car model worldwide. Therefore, the cost of redesigning a model can be reduced if the type of control can be changed in the course of development. This capability enables creative departments to produce and test novel concepts without disrupting the conventional flow of product development.
Products in development may benefit from rapid design updates, which save significant time and costs. Virtual reality allows therefore a novel, concrete, and resource-saving design evaluation method with significant application potential, since designers only need to produce the models that need to be tested, which greatly reduces time and costs [
48].
Instant feedback and design modification could further improve product quality by making it possible to detect issues at an earlier stage [
55]. In [
54], the possibility of anticipating potential problems and design changes and having precise feedback about human–machine interactions thanks to the virtual simulation, before products are produced, is discussed.
One study [
55] focused on the cost savings associated with reduced travel frequencies, since virtual reality allows the replacement of physical review meetings and interaction for immersive technical discussions.
A frequent worry when using virtual-reality-based prototypes is whether the correlation with conventional review and testing methods is maintained, as well as the degree of accuracy for usability testing.
The reviewed studies indicate that virtual-reality prototypes do not compromise correlation, i.e., virtual-reality-based usability testing applies to the same degree and may provide results equal to conventional testing with physical prototypes.
Several studies [
48,
51,
66] reported that virtual reality usability testing has a significant connection with physical testing in terms of the metrics collected. The variance of quantitative data, such as operation errors and the time spent completing tasks, was statistically analyzed, and their correlations were validated and reported with sufficient correlational data between physical and virtual prototype testing outcomes. It was found [
52] that there were no significant differences in UX questionnaire data between virtual reality and the physical prototypes in the field, but that there were correlations between rated presence in the virtual reality system and UX ratings, particularly for reported stimulation. The authors of [
48] found that the data and experience from a mixed-reality prototype vehicle were equivalent to those from a fully physical prototype vehicle. It was observed [
46] that a physical prototype automobile and a mixed-reality prototype have comparable rated usability and that there is no significant difference between the metrics they collect.
Although the authors of [
43] do not provide comparable testing methodologies for virtual systems and real-world environments, they also conclude that virtual reality might reduce costs and shorten the time taken for the virtual testing of products in development, based on metrics gathered during supported testing and interviews with participants.
3.2.8. Increased Safety for Participants
For dynamic usability testing, such as driving a car, virtual testing may provide a substantial benefit, since participants are not subjected to any actual risks. Therefore, situations such as a vehicle collision while driving, or a car accident involving passengers, would not occur in the virtual environment [
52,
66,
67].
Despite all of these benefits, virtual reality has certain drawbacks. According to the research, there are various ways in which virtual reality might impose additional constraints on conventional design reviews and usability testing processes, limiting its widespread adoption.
In fact, the technology still needs to evolve [
16] and implementing virtual reality technology may be difficult. The challenges we identified are related to a lack of realism as a result of unnatural tactile and visual senses, latency and registration issues, communication difficulties between teams, and motion sickness and other unpleasant symptoms. In the sections that follow, we analyze these problems.
3.2.9. Lack of Realism as a Result of Unnatural Tactile and Visual Senses
The difficulties of non-natural interactions between individuals and virtual prototypes owing to visual and tactile constraints were addressed frequently in the research reviewed. The complicated interactions in virtual reality have proven to be major drawbacks in industrial settings [
50].
The authors of [
15] reported that haptic feedback and multimodal interactions are still problematic and concluded that there is a lack of visualization and interaction techniques that fully harness virtual reality’s potential.
The authors of [
48] found that the movement of participants’ hands could not be well simulated, and that positioning offset occurred frequently, while [
41] identified that interacting with the geometry using the Wii Remote game control was too awkward and unnatural to fully investigate participants’ assembly inquiries. The authors of [
41] found that the collision detection experience was insufficiently robust to be helpful in their study.
Several users reported issues with virtual object feedback when compared to external device motions, such as steering-wheel twisting. Another issue mentioned was that the sense of reach and the dimensions of virtual items do not correspond to physical interaction features, such as buttons or flat surfaces that mimic multimedia screens [
46,
52].
Given the significance of natural human connection with a virtual interface, haptic devices such as gloves, suits, and others may strengthen the sensation of immersion. Including haptic technology to allow human interaction modeling of natural human senses and motion would result in a significant improvement in usability testing. In [
48], for example, it is suggested that replacing the handle with data gloves could simulate hand movement.
As a result, the effectiveness of incorporating haptic devices to improve usability testing is connected to the purpose of the testing, the complexity of the interaction, and the maturity of the virtual item under consideration. Haptic devices should be avoided for usability testing on products that are still in the early stages of development and do not have accurate hand-and-finger interaction.
The lack of realism is caused not just by touch sense issues, but also by visual sense issues. In [
46], it was observed that readability and representation in a virtual prototype system were troublesome, which the authors attributed to the head-mounted display’s low display resolution. Some participants in the tests conducted in [
46] complained that they could not view the virtual environment clearly. The study discovered that the image in the virtual environment is not fine enough, and that it is limited by the hardware devices, requiring more dynamic movement behavior and improved graphics resolution from the virtual prototype.
Due to complex interaction systems or surroundings with a substantial amount of visual information, high graphical representativeness is essential. Otherwise, only systems with a limited number of interactions may effectively correlate with conventional usability testing [
67].
A major issue is that employing virtual-reality technology without an auxiliary device, such as hand-tracking sensors, may influence user perceptions and, as a result, test findings. The adoption of a physical property, such as a flat wood table emulating the screen of a multimedia system, is one strategy for providing tactile feedback to a user engaged in a virtual environment [
66].
Each participant’s perception of a product’s depth, reach, and dimensions is spontaneous and instinctual during usability testing with physical prototypes. However, in virtual reality, the user desires touch with certain physical devices in order to handle objects in the virtual world; thus, visual calibration and positioning algorithms are necessary to modify the environment and virtual objects in order to provide an accurate user experience.
Virtual prototypes pose a challenge to the industry when human interaction that goes beyond visual verification is required. When creating functional virtual prototypes that are designed to offer visual, tactile, and aural feedback, the difficulty is to produce a high-fidelity virtual prototype that has the same features as a physical prototype. Geometric component qualities, such as high-fidelity colors and textures, part structure animations, such as when a vehicle’s door opens or when a refrigerator door handle is pulled, or even functional touchscreen displays require powerful hardware to process data within the virtual reality to ensure a reliable, immersive experience for the immersed individual.
Besides unnatural tactile and visual senses, occlusion issues also need to be addressed. It is critical to assess the participant’s emotional reaction while designing the qualitative metrics of a usability test. The authors of [
52] found that virtual surroundings diminish some sensations throughout an activity, such as user happiness or dissatisfaction. The authors discovered that while engaged in the virtual environment, the attention of some participants became predominately focused on the need to accomplish tasks. Due to the inability to watch the participants’ facial behavior, which the head-mounted display was partially hiding, the moderator did not fully observe the participants’ emotions. As a result, physical prototypes are advised for some tests in which participant behavior must be assessed by face observation [
46].
3.2.10. Latency and Registration Issues
Another issue that is commonly cited is the time between head movements and viewing the scene.
The authors of [
46] identified registration issues between virtual and physical elements of the environment, which they linked to hand and finger optical tracking controller calibration or tracking errors. The authors sought to improve tracking by instructing participants on how to adjust the head-mounted display before beginning the trial; however, the adjustment cannot be controlled from the outside by the experimenter. The resolution of the head-mounted display, according to the authors, might play a role in this scenario.
When interacting with mixed physical-virtual prototypes, several study participants mentioned a dimensional discrepancy between the virtual and physical aspects of the prototypes (for example, the cockpit’s physical wheel and the virtual air conditioning control).
The necessity of calibrating the position and visibility of virtual objects with physical prototypes was identified as a barrier throughout the research. The main difficulty is the amount of time it takes to calibrate the system, given that calibration modifications are dependent on the user’s participation.
3.2.11. Communication Difficulties between Teams
The authors of [
52] identified that virtual-reality-based design reviews of complex CAD data often suffer from communication issues between virtual reality users and team members who observe the virtual reality scene from an outside perspective (e.g., a TV screen). As a consequence, spoken descriptions are often insufficient to express a specific detail about a machinery component.
The authors of [
52] also discovered that in the virtual environment, participants voiced less affect, and [
20] found that the social exclusion of virtual reality users sharing the same physical space as colleagues during a design review session has a negative impact on communication and cooperation among team members.
3.2.12. Motion Sickness and Unpleasant Symptoms
While immersed in virtual-reality environments, several people reported unpleasant symptoms, such as nausea or headache. The source of these symptoms, according to the participants, was the lack of shadows and reflections of objects, as well as a delay in the virtual environment related to body movements and gaze [
43,
52].
Virtual-reality-based usability testing needs careful consideration of the method carried out. Immersion in a virtual world may be the first such experience in some individuals’ lives, and the strangeness of wearing a head-mounted display and handheld controllers may impair their confidence in interacting with virtual items and interfaces. When a virtual-reality-based usability testing method is designed, it is crucial to evaluate the participants’ health condition.