Abstract

In this paper, we introduce ROMOT, a RObotic 3D-MOvie Theatre, and present a case study related to driving safety. ROMOT is built with a robotic motion platform, includes multimodal devices, and supports audience-film interaction. We show the versatility of the system by means of different types of system setups and generated content that includes a first-person movie and others involving the technologies of virtual, augmented, and mixed realities. Finally, we present the results of some preliminary user tests made at the laboratory level, including the system usability scale. They give satisfactory scores for the usability of the system and the individual’s satisfaction.

1. Introduction

Driving is a complex activity that requires various skills, such as fast reactions against environmental events or unpredicted risks. Driver training and driving safety awareness in such risky cases, or with special groups of drivers (e.g., racers), can be safely done with the use of driving simulators, usually supported by virtual reality environments and motion platforms. Driving simulators are convenient for different reasons, namely: safety and risk reduction; costs reduction; greater trial availability; they cause no damage for the drivers; availability of trainers and students; possibility of recreating a variety of situations and weather conditions; possibility of repeating the same tests under the same conditions; ability to evaluate tests objectively; and the transfer of skills and training [1].

The research community has long reported on the use of driving simulators and driving-oriented virtual reality applications. Although, many of them aim at assessing or promoting driving safety, the environmental conditions, targeted users, simulators’ setups, and case studies are varied. For instance, in [2] a study was conducted regarding the fatigue among aging drivers by making use of a fixed-base driving simulator, comprising a complete automobile, fully equipped with functional pedals and a dashboard. The developed virtual reality environment was projected on a large screen reproducing a drive on a monotonous country road. Another example is shown in [3], where the targeted users are novice drivers with autism spectrum disorder. The tests were performed with a commercial simulator that displays a 210$°$ field of view on a curved screen inside an eight-foot cylinder. The simulator includes a seatbelt, dashboard, steering wheel, turn signal, gas and brake controls, right, left, side, and rear-view mirrors, as well as an adjustable seat. In [4] a study was carried out to predict motor vehicle collisions in young adults, by making use of a PC-based driving simulator. In [5] the patterns followed by drivers in adjusting the speed in curves is studied by carrying various tests in a motion-based driving simulator. In [6], how the simulator test conditions affect the severity of the simulator sickness symptoms was studied. For the tests, different simulator setups were considered, one of them considering a motion platform.

Most of the aforementioned works make use of advanced 3D graphics and sound; some also including motion platforms. However, many of them focus mainly on driving skills, not on safety awareness. In any case, the integration of other kind of multimodal stimuli, which might increase the user’s immersion, is usually not considered, and only a few works have been reported by the research community. In this regard, one of the earliest multimodal immersive systems was the Sensorama, a system patented in 1962 by Morton L. Heilig [7]. The technology integrated in the Sensorama allowed a single person to see a stereoscopic film enhanced with seat motion, vibration, stereo sound, wind, and aromas, which were triggered during the film. Recent research works dealing with immersive multimodal systems are reported in [8,9], also involving individual experiences. On the other hand, the rapid technological advancements of the last years have allowed the development of commercial solutions that integrate a variety of multimodal displays in movie theatres, such as in [10,11,12], where these systems are usually referred to as 4D or 5D cinemas or theatres. Some claim that this technology shifts the cinema experience from “watching the movie to almost living it” [13], also enhancing the cinematic experience while creating a new and contemporary version of storytelling, which can be conceptualized as a “reboot cinema” [14]. Although these systems are not centred in driving simulation, they are of interest because they include multimodal stimuli and account for a set of users rather for a single user.

In this paper, we present the outcomes of a usability test performed with ROMOT [15], a RObotized 3D-MOvie Theatre involving multimodal and multiuser experiences, for a case study in driving safety awareness. ROMOT follows the concept of a 3D movie theatre with a robotized motion platform and integrated multimodal devices, also supporting some level of audience-film interaction. ROMOT was initially conceived to be the central attraction of a driving safety awareness exhibition. However, ROMOT has been designed as a multi-purpose 3D interactive theatre, since it is highly versatile as it is prepared to support different types of setups and contexts, including films/animations or even simulations that could be related to a variety of contents, such as learning, entertainment, tourism, or even driving safety awareness, as shown in this paper. Currently, ROMOT supports and integrates various setups to fulfil different needs: first-person movies, mixed reality environments, virtual reality interactive environments, and augmented reality mirror-based scenes. The contents of all of the different setups are based on a storytelling and are seen stereoscopically, so they can be broadly referred to as 3D movies, mainly showing virtual generated graphics.

Due to the fact that ROMOT is a laboratory system that has been built from scratch (both hardware and software), differently from other commercial systems, it is highly versatile, being easily adapted to different kinds of public, purposes, contents, setups, etc. This opens new avenues in research related to e.g., HCI, robotics, learning, perception, etc. Additionally, the costs of such a laboratory system are significantly lower of that of similar commercial systems.

Despite ROMOT could be perfectly used as a classical driving simulator adding the vehicle controls (thanks to its versatility), the setups shown here do not represent a skill-oriented driving simulator to study how people perform while driving, but a 3D interactive theatre with simulated content, designed specifically to enhance and improve driving safety awareness. So far, similar systems have not been used for the purpose of driving safety awareness.

This paper is focused on the development and initial user evaluation of ROMOT, and is organized in the following way. First, we show the main technical aspects behind the construction of ROMOT and the integrated multimodal devices and interaction capabilities. It is worth mentioning that, differently from other existing commercial solutions, ROMOT accounts for a 180$°$ curved screen to enhance user immersion. Then, we show the different kind of setups and contents that were created for the case study on driving safety awareness. Finally, we show the first outcomes regarding the usability of the system and the individual’s satisfaction. To the best of our knowledge, this is the first work reporting audience experiences in such a complete system.

2. Materials and Methods

2.1. Construction of ROMOT

The house (audience) was robotized by means of a 3-DOF motion platform with capacity for 12 people (Figure 1). The seats are distributed in two rows, where the first row has five seats and the second one, seven seats. This motion platform is equipped with three SEW Eurodrive 2.2 Kw electric motors coupled with a 58.34 reduction-drive. The parallel design of the robotic manipulator alongside with the powerful 880 N·m motor-reduction set, provide a total payload of 1500 Kg, enough to withstand and move the 12 people and their seats.

The design of the robotized motion platform allows for two rotational movements (pitch and roll tilt) and one translational displacement along the vertical axis (heave displacement). The motion platform is capable of featuring two pure rotational DOF, one pure translational DOF (the vertical displacement), plus two “simulated” translational DOF by making use of the tilt-coordination technique [16] (using pitch and roll tilt to simulate low-frequency forward and lateral accelerations). Thus, it is capable of working with five DOF, the yaw rotation being the only one completely missing. It is, therefore, a good compromise between performance and cost, since it is considerably cheaper to build than a 6-DOF Stewart motion platform [17], but its performance could be similar for some applications [18]. The motion platform is controlled by self-written software using the MODBUS/TCP protocol. The software includes not only the actuators’ control but also the classical washout algorithm [19], tuned with the method described in [20].

Figure 2 shows the kinematic design of the motion platform. The 12 seats and people are placed on the motion base, which is moved by three powerful rotational motors that actuate over the robot legs. The elements in yellow transmit the rotational motion of the motors to the motion base while ensuring that the robot does not turn around the vertical axis (yaw).

The motion envelope of parallel manipulators is always a complex hyper-volume. Therefore, only the maximum linear/angular displacements for each individual DOF can be shown (see

Table 1. Motion platform excursions for each individual DOF.

	Heave (m)	Pitch (°)	Roll (°)
Minimum	−0.125	−12.89	−10.83
Maximum	+0.125	+12.89	+10.83

). Combining different DOF results in a reduction of the amount of reachable linear/angular displacement of each DOF. Nevertheless, this parallel design allows for large payloads, which was one of the key needs for this project, and fast motion [21]. In fact, the robotized motion platform is capable of performing a whole excursion in less than 1 s.

In front of the motion platform, a curved 180$°$ screen is placed, with 3 m height (and a 1.4 m high extension to display additional content) and with a radius of 3.4 m. Four projectors display a continuous scene on the screen, generated from two different camera positions to allow stereoscopy. Therefore, in order to properly see the 3D content, users need to wear 3D glasses.

Although some smaller setups introduce the display infrastructure on the motion-platform (so that they move together and inertial cues are correctly correlated with visual cues), the dimensions of ROMOT’s screen strongly recommend that the display infrastructure is kept fixed on the ground. Therefore, the visual parallax produced when the motion platform tilts, or is displaced with respect to the screen, needs to be corrected by reshaping the virtual camera properties so that the inertial and visual cues match. This introduces an additional complexity to the system, but allows the motion platform to be lighter and produce higher accelerations, increasing the motion fidelity [22].

2.2. Multimodal Displays Integrated in ROMOT

In order to enrich the experience of the users and make the filmic scenes more realistic, a set of multimodal displays was added to the robotized platform (Figure 3):

• An olfactory display. We used the Olorama wireless aromatizer [23]. It features 12 scents arranged in 12 pre-charged channels, which can be chosen and triggered by means of a UDP packet. The device is equipped with a programmable fan that spreads the scent around. Both the intensity of the chosen scent (amount of time the scent valve is open) and the amount of fan time can be programmed.
• A smoke generator. We used a Quarkpro QF-1200. It is equipped with a DMX interface, so it is possible to control and synchronize the amount of smoke from a computer, by using a DMX-USB interface such as the Enttec Open DMX USB [24].
• Air and water dispensers. A total of 12 air and water dispensers (one for each set). The water and air system was built using an air compressor, a water recipient, 12 air electro-valves, 12 water electro-valves, 24 electric relays and two Arduino Uno to be able to control the relays from the PC and open the electro-valves to spray water or produce air.
• An electric fan. This fan is controllable by means of a frequency inverter connected to one of the previous Arduino Uno devices.
• Projectors. A total of four full HD 3D projectors.
• Glasses. A total of 12 3D glasses (one for each person).
• Loudspeaker. A 5.0 loudspeaker system to produce binaural sound.
• Tablets. A total of 12 individual tablets (one for each person).
• Webcam. A stereoscopic webcam to be able to construct an augmented reality mirror-based environment.

It is important to emphasize that all of the multimodal actuators can be controlled from a computer, so that they can be synchronized with the displayed content and with the motion platform. Therefore, users are able to feel the system’s response through five of their senses:

• Sight and stereoscopy: users can see a 3D representation of the scenes on the curved screen and through the 3D glasses; they can see additional interactive content on the tablets; they can see the smoke.
• Hearing: they can hear the sound synchronized with the 3D content.
• Smell: they can smell essences. For instance, when a car crashes, they can smell the smoke. In fact, they can even feel the smoke around them.
• Touch: they can feel the touch of air and water on their bodies; they can touch the tablets.
• Kinaesthetic: they can feel the movement of the 3-DOF platform.

Apart from that, the audience can provide inputs to ROMOT through the provided tablets (one tablet per person). This interaction is integrated in the setup of the “3D-virtual reality interactive environment”, which is explained as part of the following section. It is worth noting that, instead of 3D glasses and a screen, we could have used head-mounted displays (HMD). However, the interaction with tablets could be awkward with the use of an HMD. Furthermore, the intention of ROMOT is not to isolate users but to make them feel committed to learn driving safety topics in a familiar and multi-user environment.

2.3. System Setups

Four different system setups and related stereoscopic content were elaborated, which are described in the following sub-sections. All of them were created using Unity, a powerful game engine that allows the developer to work easily with 3D elements, animations, videos, stereoscopic rendering, and even augmented reality.

2.3.1. First-Person Movie

A set of driving-related videos were recorded using two GoPro cameras to create a 3D movie set in the streets of a city. Most of the videos were filmed by attaching the GoPro cameras to a car’s hood (Figure 4) in order to create a journey with a first-person view and increase the audience’s immersive experience by locating them at the centre of the view, as if they were the protagonists of the journey.

In addition, the film includes audio consisting of ambient sounds and/or a locution that reinforces the filmed scenes. In some cases, synchronized soft platform movements or effects like a nice smell or a gentle breeze, help create the perfect ambience at each part of the movie, and make the experience more enjoyable for the audience, also providing a greater immersion in the recreated trip.

2.3.2. Mixed Reality Environment

3D video and 3D virtual content can be mixed creating a mixed reality (MR) movie that helps the audience perceive the virtual content as if it were real, making the transition from a real movie to a virtual situation easier.

In this setup, the created 3D virtual content—a 3D virtual character in the form of a traffic light—interacts with parts of the video by creating the virtual animation in such a way that it is synchronized with the contents of the recorded real scene. The virtual character talks to the audience at certain parts of the movie, giving them good advice related to correct driving in different city environments, such as crossroads. Virtual shadows of the synthetic character are also considered to make the whole scene more real (Figure 5).

2.3.3. Virtual Reality Interactive Environment

In this setup, a 3D model of a city was recreated in order to deal with situations, such as accidents, outrages, or serious violations of traffic safety norms that cannot be easily recreated in the real world. In this case, different buildings were created and merged to a map of the streets of a city. Street furniture, traffic signs, traffic lights, etc. were added too in order to create the virtual city as detailed as possible. Vehicles and pedestrians were further animated to create every situation as real as possible (Figure 6). Each driving and environmental situation was created using a storyboard that contains all of the contents, camera movements, special effects, locutions, etc. Thereby, at the end, a set of situations were derived that could be part of a movie.

In this case, the aim was not to make the audience just look at the screen and enjoy a movie, but to make them feel each situation, to be part of it, and to react to it. That is why platform movements and all of the other multimodal displays are so important.

When each situation takes place, the audience can feel that they are in the car driving thanks to the platform movements that simulate the behaviour of a real car (accelerations, decelerations, turns, etc.). In some of the scenes, the recreated movie pauses and asks the audience for their collaboration in two ways: either a question is shown to the audience for them to give an answer, or they are asked to contribute to driving, by accelerating or decelerating at certain points of the trip (Figure 7). For instance, in the first case, the individual tablets vibrate and a question appears, giving the users some time to answer it by selecting one of the possible options. When the time is up, they are prompted to report whether the answer was correct or not, and the virtual situation resumes, showing the consequences of choosing a right or a wrong decision. Crashes, outrages, rollovers, etc., the audience can feel, in first person, the consequences of having an accident thanks to the robotic platform movements and the rest of the multimodal feedback, such as smoke, smells, etc.

Each correct answer increases the individual score at each of the tablets. When the deployed situation finishes, the audience can see the final score on the large, curved screen. The people having the highest score are the winners, who are somehow rewarded by the system by receiving a special visit, a 3D virtual character that congratulates them for their safe driving (see the next sub-section).

2.3.4. Augmented Reality Mirror-Based Scene

This setup consists of a video-based augmented reality mirror (ARM) [25,26] scene, which is also seen stereoscopically. ARMs can bring a further step in user immersion, as the audience can actually see a real-time image of themselves and feel part of the created environment. The AR scenario is built of a mixture of a stereoscopic image of the audience, which is captured in real-time by the integrated stereoscopic webcam, blended with a 3D virtual character and a 3D virtual scenario (in this case a 3D model of the platform and the seats). The scene works as a mirror, as the audience sees their own image in front of them, correctly aligned as if it was an image reflected by a mirror. The 3D virtual scenario creates a strong integration of the stereoscopic image from the webcam and the 3D character by using occlusion (for example when the 3D virtual character is walking behind a seat). An example of the ROMOT’s ARM is shown in Figure 8, where the 3D virtual character is seen at the left bottom part of the image. In this case, only two users are sitting down (upper seats) as the image was taken at the laboratory level.

This ARM environment is used in the final scenes of the aforementioned virtual reality interactive environment (previous sub-section), where the user(s) with the highest score is/are rewarded by a virtual 3D character that walks towards him/her/them. Together with this action, virtual confetti and coloured stars appear on the environment, accompanied with winning music that includes applauses.

3. Results

In this section, we present the results of some preliminary user tests made at the laboratory level (Figure 9). A total of 14 people tested the system and participated in its evaluation, which consisted on filling out a pair of questionnaires that were related to the usability of the system and the individual’s satisfaction, where the system usability scale (SUS) [27] was chosen to measure usability. These kind of tests are commonly used to derive quantitative evaluation of technological systems regarding to usability, and SUS has become a standard. Some recent works using these tests can be found in [28,29,30].

The participants were some of the research staff of the IRTIC Institute, where seven of them were women and seven were men. Only three of the participants contributed to the present research work at some of their stages. Though this might not be considered as a full objective audience, it can give us a good first notion for a user-related evaluation of the system.

The results of the SUS questionnaire are listed in

Table 2. Results of the SUS questionnaire (mean, standard deviation, minimum, and maximum).

Questions	Mean	S.d.	Min	Max
1. I think that I would like to use this system frequently	3.00	0.89	1	4
2. I found the system unnecessarily complex	0.70	0.78	0	2
3. I thought the system was easy to use	3.50	0.5	3	4
4. I think that I would need the support of a technical person to be able to use this system	1.30	1	0	3
5. I found the various functions in this system were well integrated	3.60	0.66	2	4
6. I thought there was too much inconsistency in this system	0.40	0.49	0	1
7. I would imagine that most people would learn to use this system very quickly	3.60	0.49	3	4
8. I found the system very cumbersome to use	0.60	0.8	0	2
9. I felt very confident using the system	3.30	0.64	2	4
10. I needed to learn a lot of things before I could get going with this system	0.30	0.46	0	1
SUS score	84.25

. In questions 1 to 10 the range 0–4 means: 0: strongly disagree, 4: strongly agree. The values of the SUS score, however, range from 0 to 100, meaning 100 is the best imaginable result. In the case of the ROMOT evaluation, this score reaches 84.25 points, which can be considered excellent on the scale of scores provided by the questionnaire and taking into account the fact that a minimum score of 68 would be deemed acceptable for a tool [31,32].

The results of the individuals’ satisfaction questionnaires are given in

Table 3. Results of the individuals’ satisfaction questionnaire (mean, standard deviation, minimum, and maximum).

Questions	Mean	S.d.	Min	Max
1. Overall, I liked very much using ROMOT	3.14	0.74	2	4
2. I find it very easy to engage with the multimodal content	3.29	0.8	2	4
3. I enjoyed watching the 3D movies	3.29	0.8	1	4
4. The audio was very well integrated with the 3D movies	3.43	0.73	2	4
5. The smoke was very well integrated in the virtual reality interactive environment	2.71	0.88	1	4
6. The smell was very well integrated in the virtual reality interactive environment	2.93	1.1	0	4
7. The air and water were very well integrated in the virtual reality interactive environment	2.79	1.01	0	4
8. The movement of the platform was very well synchronized with the movies	3.36	0.72	2	4
9. The interaction with the tablet was very intuitive	3.07	0.8	2	4
10. I didn’t feel sick after using ROMOT	2.64	1.23	0	4
11. I would like to use again ROMOT	3.36	0.61	2	4
12. I would like to recommend others to use ROMOT	3.64	0.48	3	4

. The scores also range from 0 to 4, meaning: 0: strongly disagree, 4: strongly agree. As it can be seen, results are quite satisfactory as eight out of the 12 mean scores are over 3 points and none are under 2.5 points. The lowest mean score belongs to question 10: “I didn’t feel sick after using the ROMOT”, which is an issue difficult to tackle, as simulator sickness highly depends on the individual. On the other hand, the highest mean score belongs to question 12: “I would like to recommend others to use ROMOT”, meaning that, overall, the individuals are satisfied with the system.

In addition to the tests, we have asked the participants to give us any other kind of feedback in a section dedicated to personal comments. We collected feedback from three of the participants. Two of them coincided in that the system was “very awesome” and they enjoyed the multimodal experience. The third one also addressed that the system provided him/her with an impressive experience and the sense of immersion was very high, as he/she could really feel like driving a car through the different scenarios and situations. However, he/she felt a little bit sick after a while, due to the combination of the stereoscopic glasses and the motion platform movements. Nevertheless, he/she still recommended the experience to others, as it was “very thrilling”.

4. Discussion and Further Work

As explained in the previous section, the preliminary quantitative evaluation of ROMOT related to usability was very satisfactory. After the tests were performed in the lab environment, our system was sent to a driving safety awareness exhibition in the Middle East. The exhibition consisted of a set of interactive applications, which included ROMOT as the central attraction (Figure 10). ROMOT and the rest of applications were designed and implemented entirely by the IRTIC institute for the same purpose, i.e., to teach and train on basic driving safety rules and transfer awareness on the importance of driving safely. In this sense, the set of interactive applications can be considered as a single learning/training tool.

As a further work, we intend to evaluate the learning and training capabilities of the whole exhibition. In order to collect some quantitative indicators, the visitors of the exhibition are asked to fulfil an online questionnaire before (ex-ante) and after (ex-post) the visit. Additionally, we are measuring the time that each individual expends at each application, the individual performance and scores obtained at each application (if any), and the trail each individual follows in the exhibition (e.g., in what order they visit the applications, if they repeat the experience, etc.). We are able to collect these data as each individual is given a unique code at the ticket entrance, which is required to activate each of the applications.

From a technological standpoint, the authors also propose and suggest testing other possibilities, such as different types of stereoscopic displays, a 6-DOF motion platform, and explore other user interaction systems different from tactile tablets. However, these kinds of improvements could raise the cost of the solution.

5. Conclusions

In this paper, we have presented the construction and first user evaluation of ROMOT, a robotized 3D-movie theatre. The work shown in this paper relates to the enhancement of audience experiences when integrating multimodal stimuli and making it interactive. As well, we show the versatility of the system by means of the different kind of generated content.

Both the setups and the film content of ROMOT can be changed for different types of user experiences. Here we have shown different setups for content related to driving safety awareness, though other filmic contents could be used, including some related to learning, training, entertainment, etc. As for the different setups, we have shown a first-person movie and others related to the technologies of virtual, augmented and mixed realities.

The outcomes regarding the usability of the system and the individual’s satisfaction are very promising, though we are aware that the system has only been evaluated at the laboratory level. It is also worth mentioning that, although there exist different commercial solutions (e.g., 4D/5D cinemas), we have not found complete research works dealing with the construction and audience evaluation of such systems.

As a further work, we intend to evaluate the developed system in a real environment (the exhibition space) with the participation of hundreds of visitors. We would also like to create more film content in order to make use of ROMOT for other purposes.