BARI: An Affordable Brain-Augmented Reality Interface to Support Human–Robot Collaboration in Assembly Tasks

Abstract

Human–robot collaboration (HRC) is a new and challenging discipline that plays a key role in Industry 4.0. Digital transformation of industrial plants aims to introduce flexible production lines able to adapt to different products quickly. In this scenario, HRC can be a booster to support flexible manufacturing, thus introducing new interaction paradigms between humans and machines. Augmented reality (AR) can convey much important information to users: for instance, information related to the status and the intention of the robot/machine the user is collaborating with. On the other hand, traditional input interfaces based on physical devices, gestures, and voice might be precluded in industrial environments. Brain–computer interfaces (BCIs) can be profitably used with AR devices to provide technicians solutions to effectively collaborate with robots. This paper introduces a novel BCI–AR user interface based on the NextMind and the Microsoft Hololens 2. Compared to traditional BCI interfaces, the NextMind provides an intuitive selection mechanism based on visual cortex signals. This interaction paradigm is exploited to guide a collaborative robotic arm for a pick and place selection task. Since the ergonomic design of the NextMind allows its use in combination with the Hololens 2, users can visualize through AR the different parts composing the artifact to be assembled, the visual elements used by the NextMind to enable the selections, and the robot status. In this way, users’ hands are always free, and the focus can be always on the objects to be assembled. Finally, user tests are performed to evaluate the proposed system, assessing both its usability and the task’s workload; preliminary results are very encouraging, and the proposed solution can be considered a starting point to design and develop affordable hybrid-augmented interfaces to foster real-time human–robot collaboration.

1. Introduction

Human–robot collaboration (HRC) is a branch of human–computer interaction (HCI); HRC investigates how humans and robots can interact in different situations. Collaboration can involve service and industrial robots: collaborative robots are usually called cobots. In particular, the interaction between technicians and cobots in an industrial context is deeply related to Industry 4.0, which is quickly changing the nature of factories in order to foster more flexible and efficient manufacturing.

Different levels of “collaborative work” can be identified [1]: synchronized (human and robot work in the same space but at different times), coexistence (human and robot are in the same environment but do not interact), cooperation (human and robot work in the same space at the same time), and collaboration (human and robot execute a task together). This paper focuses on the collaborative level and considers a human–robot collaborative assembly task (as it represents one of the more profitable HRC forms).

The design of collaborative environments is one of the main goals of HRC. The collaboration between humans and machines involves two dimensions [2]: know-how and know-how to cooperate. The first dimension is related to the capability of humans and machines to control a process, whereas the second one concerns the capability to interact with other “agents” (the term agent can denote both humans and machines/cobots). In a collaborative environment, humans and cobots exchange information by means of interfaces [3] that should also allow agents to recognize each other’s intentions to effectively and safely address the common final goal [4].

Augmented reality (AR) superimposes digital assets (e.g., 3D models and animations) into the real world. Manufacturing and repair have been immediately identified as potential application fields for AR [5]; moreover, AR has been selected as one of the nine pillars of Industry 4.0. With these premises, it is not surprising that AR has often been used to convey cobot intentions and status to human operators. Interested readers can find in [6] a systematic literature review of AR interfaces for HRC.

On the other hand, several different technologies have been considered to convey inputs from humans to cobots. Gesture, speech, and haptic are just some of the possible interfaces [7]. Each of these technologies shows some limitations; for instance, noisy environments (such as industrial ones) may adversely affect the performance of speech interfaces, whereas gestures can be limited when human operators execute assembly/maintenance tasks that require both hands. Further, haptic interfaces can be challenging to adapt to human operators’ intentions.

Brain–computer interfaces (BCIs) may overcome these issues; the following five stages implement a BCI: signal acquisition, preprocessing, feature extraction, classification, and output/control [8]. A BCI enables humans to interact with devices/machines without the involvement of peripheral nerves and muscles, thus “only” using control signals generated from electroencephalographic activity. A survey of BCIs can be found in [9].

This paper presents a novel system that uses an AR interface to convey information about the cobot’s intentions to the user and a BCI to provide the cobot (in this case, a robotic arm) commands to pick the parts of a vise and place them close to the user, thus allowing them to be assembled as the final object. The Microsoft HoloLens 2 [10], a wearable see-through head-mounted display (HMD) that enables users to display and interact with computer-generated objects, was used to implement the so-called mixed-reality paradigm. On the other hand, the NextMind [11] device was used to implement the BCI. The NextMind sensor uses EEG technology to detect neural activity from the visual cortex. Machine learning algorithms translate signals into digital commands in real-time. Since the NextMind differs from the traditional BCI in focusing on signals coming from the visual cortex, there is room for evaluating its effectiveness for human/robot collaborative tasks.

User tests were also performed to assess the system’s usability; ten people were involved in tests and an average score of 75.75 was obtained by considering the System Usability Scale (SUS) questionnaire [12]. Further, the NASA-TLX questionnaire [13] was used to assess the global user workload and, as can be expected, the highest scores were related to mental demand.

The paper is organized as follows: Section 2 reviews both augmented reality and brain interfaces to support human–robot collaboration, whereas Section 3 presents the design and implementation details of the proposed solution as well as the tasks considered for tests. Section 4 shows preliminary usability scores; results are analyzed and discussed in Section 5.

2. Background

The design of AR interfaces has been investigated since the first years of this century [14]. Several design approaches have been used to support users in different AR contexts. Industrial AR apps should be intuitive, robust, evocative, and usable, as outlined in [15].

A recent and detailed analysis of the state-of-the-art related to AR-enhanced interaction between humans and robots can be found in [16]. The authors categorized augmentation into two different categories: on robots and surroundings. The first approach adds information on top of the robot, whereas augmentation of surroundings uses walls, floors, and other physical objects around the robot to anchor computer-generated information. The authors outline how visual augmentation may provide the following benefits:

• Facilitated robot programming;
• Real-time support for control and navigation;
• Improved safety;
• Communication of the robot’s intention;
• Increased expressiveness.

Moreover, twelve main application domains have been identified: domestic and everyday use, industrial applications, entertainment, education and training, social interaction, design and creative tasks, medical and health, telepresence and remote collaboration, mobility and transportation, search and rescue, robots for workspaces, and data physicalization. The proposed work ranks in the industry application domain.

AR interfaces have been deeply investigated to support HRC effectively, as well as the use of BCIs to control robots. An EEG-based BCI has been used to command a semi-autonomous robotic arm by means of motor imagery (MI); in particular, a way to optimize the on-line classification accuracy of an MI-based BCI system by choosing an optimal time window for calculating common spatial patterns (CSP) was proposed in [17]. Functional magnetic resonance imaging (FMRI) has been used as an input device to identify the user’s intentions and convert them into actions performed by a humanoid robot [18]. Another work based on the classification of the P300 waveform is proposed in [19]. The P300 waveform is affected by visual stimuli, and a six-LED-based stimulator was developed in [19]; couples of LEDs were placed at the end of transparent sticks, which represent directions in the Cartesian space. The user can focus on the visual stimuli and control a Mico Kinova robotic arm according to correspondences among LEDs and commands; for instance, LED 1 is used to convey a stimulus to move the arm forward. A survey of BCIs for humanoid robot control can be found in [20].

Hybrid interfaces, which use both AR and BCI, are also known in the literature. The first attempts to propose AR–BCI-based interfaces can be dated back to 2010 [21,22]. In [21], the BCI is based on P300 waveform analysis, and visual stimuli are six cubes on which are placed unique ARToolkit markers [23]; the AR interface recognizes these markers and augments each cube with a number placed on it. Each number codes well-defined positions that a Kuka robot arm uses to pick and place objects. Further, tests involving four users were performed to evaluate system usability. In [22], integration between BCIs based on the steady-state visual evoked potential (SSVEP) and 3D environments is evaluated. A thorough study about BCI in augmented and virtual spaces was later presented in [24].

Commands of interest are displayed on an HMD also in [25]; commands are overlaid in 3D with respect to the perspective view of the robot, which has to perform both a pick and place and a gaming task. The pick and place task was also supported in [26]. Eye tracking allows the user to select the object to pick, and the BCI is used to confirm object selection, trigger switching of an action sequence, and control the aperture and height of the gripper. The AR interface provides users feedback during the grasping and lifting processes, thus efficiently avoiding obstacles. The feasibility of combining AR and BCI technologies has been assessed in [27]; in particular, the authors focused on BCI based on SSVEP and optical see-through augmented reality. The influence of external and internal noise on BCI accuracy was evaluated, and an extended design space for target integration in AR–SSVEP-based applications was also proposed.

Further, mobile robots such as automated/automatic guided vehicles AGVs, drones, and wheelchairs can be controlled by hybrid AR–BCI interfaces; for instance, a drone is guided in [28], and different kinds of BCIs are investigated in [29] to train users to control wheelchairs. In this case, the authors show how the BCI can be effectively combined both with AR and virtual reality (VR) interfaces. AR and VR interfaces provided the same results in terms of usability and user experience.

All the above-mentioned works detect brain activity by means of “invasive” (and sometimes expensive) devices such as neuroheadsets, and they can be considered laboratory experiments. On the other hand, the proposed solution is based on the NextMind [11]. This affordable and ergonomic device can be easily and comfortably used with HMD AR devices such as the Microsoft HoloLens. The proposed solution can be profitably employed in real work scenarios, and the usability tests (see Section 4) clearly outline users’ appreciation.

The NextMind is a wireless device that can be paired with other devices supporting the Bluetooth protocol. It can be attached by a headband, and it should be placed at the back of the head, where the visual cortex is located, with its lower part over the inion. Nine electrodes, which should always be in contact with the scalp, detect brain activity. The NextMind core is mainly built around two components: NeuroTags and NeuroManager. NeuroTags are identified by blinking/flickering textures (different NeuroTags are related to textures of blinking at different frequencies) that make any object of the scene “mind-controllable”; a NeuroTag represents a unique visual signature that generates a response in the user’s brain. On the other hand, NeuroManager manages the communication between the NeuroTags and the NextMind engine, which is in charge of processing signals received from the electrodes. No more than 10 NeuroTags should be inserted in the same scene; moreover, the NextMind automatically disables the NeuroTags outside the user’s field of view. Figure 1 shows an example of a NeuroTag associated with a flickering texture. The three green segments move to compose a triangle when the visual signature has been detected; this is the standard feedback provided by the NextMind. The NextMind has been used to compare selection performance between gestures and BCI in AR interfaces [30]; results clearly outline the potential of this device.

Contrary to the presented state-of-the-art, the NextMind focuses only on visual cortex signals, providing an interesting and intuitive selection mechanism that is assessed in this paper to verify its effectiveness for human–robot collaborative tasks.

3. Material and Methods

This section presents HW and SW system architecture, the user interface (UI), and the selected tasks for tests.

3.1. Hardware Architecture

From the hardware point of view, the proposed solution is depicted in Figure 2. The NextMind is connected to the HoloLens 2 via Bluetooth, whereas the AR device is connected to a LAN by Wi-Fi. The HoloLens, the COMAU e.DO manipulator [31], and a control PC are connected to the same LAN as the HoloLens. Even if e.Do is not a real cobot, its plastic cover and limited speed allow humans to share the same workspace without danger, thus simulating the behavior of a cobot. A control PC (also named terminal) is introduced in order to decouple the interface by the selected robot; in this way, the proposed solution is robot-independent, as all selection commands are sent to a server running at the control PC (see Section 3.2).

In the same way, robot feedback is sent back to the terminal then processed and presented to the user by the AR interface.

3.2. Software Layers

Software layers are shown in Figure 3. An application designed by the NextMind SDK receives and processes EEG data. This application can be fully integrated with the Unity3D game engine [32]; in this way, the results (feedback) of EEG data processing can be directly managed within the AR application deployed for the HoloLens 2. The AR interface allows the user to select the vise components (see Section 3.3) by means of the associated NeuroTags.

When a component is selected, the corresponding command is sent to the control PC. A server written in Python is running at the terminal; this server exposes a number of functions corresponding to the number of the vise parts plus some other control functions to move the robotic arm in some specific positions (for instance, when the robot is waiting for a new pick and place command). Each vise component is placed in a predefined position in the workspace (see Figure 4). Therefore, each pick function refers to a predefined position and orientation (6DOF) of the gripper. The e.Do returns a status message to the control PC after executing each command; this feedback is processed and forwarded to the user interface. For instance, a warning animation is displayed and a new selection is impossible when the arm moves.

The Unity application is organized in three scenes: main menu, calibration, and project. The main menu appears when the application is started, and it allows the user to select the other two scenes. A gaze-tracking constraint is activated for the main menu scene; in this way, the selection panel can always be displayed in front of the user. The calibration scene allows the user to calibrate the NextMind device. A graphical representation denotes the quality of signals received from the electrodes. Moreover, the user can focus on a set of NeuroTags for training. The project scene allows the user to perform selected tasks (see Section 3.4).

3.3. User Interface

When the project scene starts, the user can see the vise either assembled or exploded (the assembled view of the vise is displayed on a virtual plane the user defines at the beginning). In particular, due to the limitation of ten NeuroTags per scene, the vise is divided into three different sub-parts/sections (for the sake of readability, the three sections are bounded in rectangles; see Figure 5). Supplementary Material (a video) shows the proposed UI.

Each section of the model is associated with a NeuroTag. When a section is selected, all the corresponding parts are displayed; two NeuroTags are associated with each part: Explore and Select (see Figure 6). The Explore mode allows the user to: achieve more information about the selected component, change the representation from solid to wireframe, rotate the object, select the object (e.Do will pick the real object), see a video about the object, and skip back to the visualization of all the section components (see Figure 7). The Explore mode is needed to complete Task 1 (see Section 3.4). On the other hand, the Select mode directly activates the robot to pick the selected objects, and it is mainly used to complete Task 2.

In order to provide more efficient feedback than the standard three green lines forming a triangle when the user is focusing on a NeuroTag, a growing circle is associated with the selection mechanism. When the green circle grows up to the size of the white circle (this means that the user focused on the NeuroTag for a given time, e.g., two seconds), the NeuroTag triggers the selection (see Figure 8). It is worthwhile to note that this mechanism prevents unwanted selections.

3.4. Tasks

Two tasks related to a vise were identified; the vise project was downloaded from [33], and all parts were 3D printed. The two tasks were:

• Research of information: Users had to visually identify and select a specified component of the vise in the AR interface (in particular, in the exploded view of the vise) and select its info panel. Users had to identify the weight and size of the component;
• Assembly of the vise: Users had to select all the components in a well-defined sequence using the corresponding NeuroTags. After selections, e.DO was in charge of picking components and placing them in a pre-set position, thus allowing the users to assemble the vise.

Elements were placed in fixed positions within the robot workspace, as shown in Figure 4; a foam support contained the vise parts in order to keep them steady when the gripper closed to pick. Figure 4 also shows the area where components were released.

4. System Evaluation

To the best of the authors’ knowledge, this is the only work that uses the NextMind to design and develop a BCI–AR interface to foster human–robot collaboration. The NextMind is an affordable and ergonomic device designed to be easily and comfortably used together with AR headsets, such as the Microsoft HoloLens. Moreover, the NextMind does not require any kind of waveform analysis since it provides an intuitive selection mechanism based on visual cortex signals that are automatically detected by the device APIs. Therefore, other BCI interfaces were not considered for comparison. Instead, we focused on evaluating the usability of the proposed system in order to assess limitations and users’ appreciation for an industrial assembly task.

4.1. User Test

Preliminary tests involved 10 users aged between 22 and 29. Most of them had previous knowledge of AR technologies, but none had tried a BCI before. All users were volunteers and did not receive any reward. Tests were performed according to the following steps:

• Welcome and device sanitization;
• Device setup (users first wore the NextMind and then the HoloLens 2);
• Calibration (the NextMind has to be calibrated in order to obtain the best performance; this step is also needed to check the contact point between the electrodes and the scalp);
• Training (a simple tutorial application was developed in order to familiarize users with the BCI);
• Virtual plane definition (users had to define, by gestures, a virtual plane where the vise is represented when assembled);
• Operative tasks (users performed the two tasks described in Section 3.4);
• Compilation of two questionnaires (the SUS questionnaire was used to assess usability, whereas the NASA-TLX evaluated the global workload).

4.2. Results

Table 1. SUS scores.

User	SUS Score	Grade	Adjective Rating
1	62.5	D	Okay
2	67.5	D	Okay
3	92.5	A+	Best Imaginable
4	77.5	B+	Good
5	72.5	C+	Good
6	75.0	B	Good
7	82.5	A	Excellent
8	70.0	C	Good
9	77.5	B+	Good
10	80.0	A−	Good
Mean	75.75	B	Good
Standard Deviation	7.98

shows the results obtained from the SUS questionnaire in terms of SUS score, grade, and adjective rating for each user; mean values and standard deviation are depicted at the bottom of the table.

Pertaining to the NASA-TLX questionnaire,

Table 2. NASA-TLX results for Task 1.

Users	Individual Scores (Weighted)
	Mental	Physical	Temporal	Performance	Effort	Frustration	Mean
1	70	40	150	120	210	60	43.33
2	135	135	300	25	150	0	49.67
3	105	10	180	20	45	0	24.00
4	0	60	60	45	45	45	17.00
5	45	70	60	135	30	50	26.00
6	50	0	260	200	120	60	46.00
7	350	0	240	45	180	5	54.67
8	325	0	90	80	225	35	50.33
9	260	150	45	125	130	0	47.33
10	110	10	105	175	150	20	38.00
	Diagnostic Subscores						Overall
GSR	161.11	67.86	149.00	97.00	128.50	39,29	39.63

and

Table 3. NASA-TLX results for Task 2.

Users	Individual Scores (Weighted)
	Mental	Physical	Temporal	Performance	Effort	Frustration	Mean
1	130	20	120	120	130	130	43.33
2	90	300	320	50	225	0	65.67
3	105	5	140	25	40	0	21.00
4	40	140	120	100	100	0	33.33
5	195	0	340	320	135	30	68.00
6	80	0	40	250	60	30	30.67
7	285	0	380	15	500	70	83.33
8	375	0	120	100	195	40	55.33
9	350	0	40	120	260	110	58.67
10	0	5	180	120	255	240	53.33
	Diagnostic Subscores						Overall
GSR	183.33	94.00	180.00	122.00	190.00	92.86	51.27

display for the two proposed tasks, respectively, the individual weighted score for each parameter provided by the users, as well as the mean value, which represents the task load index scored by each user. At the bottom of the table, the weighted group score results (GPR) are provided as well, both in terms of individual diagnostic subscores for each parameter and overall task load index.

Completion times for the two tasks were also measured (see

Table 4. Completion times, in minutes, for the two tasks (T1 and T2, respectively).

User	1	2	3	4	5	6	7	8	9	10	Mean
T1	04:05	00:57	00:55	00:45	01:39	00:30	01:03	00:43	00:23	00:37	01:10
T2	17:36	14:02	12:57	13:02	13:04	12:08	11:56	12:21	11:14	12:02	13:02

).

5. Discussion

This section presents the results analysis along with the current limitations and possible future works.

5.1. Results Analysis

The SUS questionnaire score displayed in Table 1 shows that the average SUS score is 75.75, which means good usability according to the adjective rating scale proposed by Bangor et al. in [34] and more recently updated by Sauro [35]. Only one user (# 1) provided an evaluation of 62.5 (still considered OK on the corresponding adjective scale), below the average SUS score, which is 68; on the other hand, three users (# 3, 7, and 10) provided a score equal or over 80, which correspond to an A on the school-grade scale; more specifically, user # 7 provided a score of 82.5, which correspond to an excellent usability, whereas user # 3 provided a score of 92.5, which corresponds to best imaginable.

Concerning the NASA-TLX questionnaire, the overall weighted Global Workload Score provides a value which represents the Task Load Index of a given task on a scale from 0 to 100. Based on [36], a score of 39.63 for Task 1 is below the average value of 48.74, whereas a score of 51.27 for Task 2 is slightly above the average value. Referring to the diagnostic sub-scores, the highest scores are related to temporal and mental demand for Task 1; on the other hand, the lowest scores refer to physical demand. Further, frustration received low ratings, thus confirming the good usability of the proposed solution. The high mental demand score is strictly related to the nature of the interface, which requires focus and time to activate the NeuroTags associated with each action. Similar results were obtained for Task 2, as effort, mental, and temporal demand are the highest scores. Overall, the temporal demand is slightly higher for Task 2 compared to Task 1, but it does not increase very much compared to the actual average completion time, which increases from 70 s to more than 13 min. Mental demand is slightly higher for Task 2 than for Task 1, whereas the task is clearly more demanding (research of information vs. assembling a vise). According to the differences between the two tasks, the perceived effort for Task 2 is more significant than for Task 1.

User # 1 obtained the worst performance both from a subjective point of view and in terms of completion time. The most relevant problem observed while performing the tests was that user # 1 had great difficulty selecting NeuroTags; this can be due to poor calibration or to some distracting elements in the user’s field of view. For instance, a monitor was used to indicate to users the exact selection sequence; displayed images might interfere with the AR interface, thus reducing the user’s capability to focus on the NeuroTags. Similarly, user # 10 reported high mental load and low levels of frustration for Task 1 but null mental load and considerably high levels of frustration for Task 2. It is possible that the NextMind was not fixed properly, and the electrodes could not detect the visual stimuli.

5.2. Limitations

The development and assessment of the proposed system were crucial steps to understand the existing limitations. Some limitations were related to the NextMind device and are depicted in the device user’s guide, whereas others were assessed through the usability test. First of all, the suggested maximum number of NeuroTags to be used simultaneously is 10: the NextMind is not able to handle more than 10 different NeuroTags; otherwise, multiple actions will be mapped to the same NeuroTag, and the number of false positive activations will drastically increase. Moreover, the interaction paradigm is limited to a button approach. Activating a command on each activation of the corresponding NeuroTag leads to two options: a NeuroTag can either directly control a specific action/command if the application has a limited set of actions available at the time (fewer than 10); otherwise, a NeuroTag is used to navigate and select actions among a scrollable menu or other similar solutions. Another limitation related to the wearability of the device is that people with long hair may have difficulty obtaining a good calibration score. Moreover, the surrounding environment may affect system usability, since visual distracting elements may undermine the user’s capability to focus on the NeuroTags and effectively activate the available actions. Finally, the user interface design plays a key role, and parameters such as the number of visible NeuroTags and NeuroTag dimension and positions deeply affect system usability, as further investigated in [37].

5.3. Improvements and Future Work

Further tests should clarify how much poor calibration and distracting elements in the real environment impact system usability and provide best practices to improve user interface design. Moreover, other selection methods have to be compared with the NeuroTag-based one: since the HoloLens 2 also supports selections by gestures and gaze tracking, future research will extend the proposed interface considering these selection methods and comparing them with inputs by brain activity.

Although the interface was evaluated in an industrial scenario, alternative robotic domains could benefit from it. As shown by the corresponding scientific literature, service robotics would be the ideal candidate for the proposed AR–BCI interface [38,39]. Specifically, people with disabilities could control a robotic arm by using the NextMind device. NeuroTags might be projected into the real world, thus eliminating the need for the user to wear any device. Depending on the specific disability, the AR–BCI interface could be compared with different input interfaces (e.g., gaze, gestures, etc.) to assess and verify its effectiveness.

6. Conclusions

This paper presents BARI, a novel hybrid brain and AR interface to foster human–robot collaboration in assembly tasks. AR is used to convey to the user both information related to the robot status and graphical representations of the parts to be assembled; on the other hand, a BCI is used to give the robot commands to pick selected objects. Since the NextMind differs from traditional BCI interfaces by focusing only on signals coming from the visual cortex, it provides an interesting and intuitive selection mechanism that has been assessed to verify its effectiveness for human–robot collaborative tasks. An evaluation involving ten users confirmed the potential of these hybrid interfaces to support HRC.

Although obtained results can only be considered as preliminary, and more tests are necessary to validate the system, good levels of usability are the essential prerequisite for further investigation. Overall, the users evaluated the experience positively, and the system could be effectively deployed in industrial assembly scenarios.

The usability evaluation proved that poor calibration or distracting elements in the real environment might interfere with the proposed system. Further tests should clarify how much these factors impact system usability and may suggest best practices for the user interface design. Alternative selection methods such as gestures and gaze tracking will also be compared with inputs by brain activity. Finally, it is possible to apply the proposed interface to other robotic domains, such as service robotics.