A Pilot Study of Game Design in the Unity Environment as an Example of the Use of Neurogaming on the Basis of Brain–Computer Interface Technology to Improve Concentration

Abstract

The article describes the practical use of Unity technology in neurogaming. For this purpose, the article describes Unity technology and brain–computer interface (BCI) technology based on the Emotiv EPOC + NeuroHeadset device. The process of creating the game world and the test results for the use of a device based on the BCI as a control interface for the created game are also presented. The game was created in the Unity graphics engine and the Visual Studio environment in C#. The game presented in the article is called “NeuroBall” due to the player’s object, which is a big red ball. The game will require full focus to make the ball move. The game will aim to improve the concentration and training of the user’s brain in a user-friendly environment. Through neurogaming, it will be possible to exercise and train a healthy brain, as well as diagnose and treat various symptoms of brain disorders. The project was entirely created in the Unity graphics engine in Unity version 2020.1.

1. Introduction

The gaming industry has grown in unimaginable ways over the past few years. The era of computer games began with a game created in 1947 by Thomas T. Goldsmith Jr. and Raya Manna, called Estle, which simulates a missile hitting a designated target. The control was carried out with the help of special knobs that changed the speed and angle of the projectile. The computer games market gained its greatest momentum in the 1980s, where many publishers and the most popular arcade games of the time, such as ‘De-fender’ or ‘3D Monster Maze’, were created. Over time, publishing companies began to join forces, and the industry was fed by an increasing budget, thanks to which many RTS games such as ‘The Sims’ by Maxis and other online games began to appear. All these events, to a large extent, led to a significant increase in interest in computer games. Nowadays, there are many development environments on the market that allow us to create various games. The Unity graphics engine is one of the best known [1]. Unity is a toolkit for creating three-dimensional (3D) and two-dimensional (2D) games that was first introduced at a conference (Apple’s Worldwide Developers Conference) in 2005.

The brain–computer interface is a technology that allows direct contact between the brain and an external device. The signal analysis of electroencephalograms (EEGs) has contributed to the research on BCIs. The beginnings of EEGs date back to 1929 when doctor Hans Berger first registered the activity of the human brain using an electroencephalograph [2]. Thanks to the initiation of research on EEGs, it is now possible to construct modern BCI interfaces. Noninvasive methods of neuroimaging have developed relatively recently and are divided into the following areas: positron emission tomography (PET), magnetoencephalography (MEG), and magnetic resonance imaging (MRI). In contrast, neurogaming uses brain waves to influence the game without the use of a keyboard or mouse. Recently, many companies have started using EEGs to create BCI interfaces, for example, NeuroSky, Emotiv Systems Inc., and OpenBCI. The BCI is a young technology that keeps getting more and more interest [3]. Along with the development of brain–computer technology, more and more algorithms have appeared, allowing the spatial filtering method for feature extraction in motor imagery (MI)-based BCIs. These methods have been continuously developed, including, for example, a fusion algorithm based on the Dempster-Shafer theory, which takes into consideration the distribution of features [4]. When carrying out research in the field of EEGs, one should bear in mind the accompanying disruptions. Thus, artifacts play a significant role in the implementation of brain–computer technology. At cocktail parties, our brains often receive simultaneously visual and auditory information. Therefore, scientists have explored the effects of audiovisual inputs in a simulated cocktail party [5].

This article presents the use of neurogaming using the Emotiv Headset to create a computer game. Moving the player will be via the BCI; we will be able to move forward, backward, right, and left. Using the cursor in the main menu, the ability to look around the game world will also be possible thanks to the brain–computer interface. An innovative element that will allow us to navigate in the game is the Emotiv Epoc + NeuroHeadset, with the software provided by the Emotiv company. The Unity engine in the Unity 2020.1 version was used to create the game, owing to the transparency and simplicity of using the environment.

2. Materials and Methods

2.1. Unity Environment

As already presented in the introduction to the article, the Unity engine allows us to create 2D and 3D games. The first version of Unity, released in 2005, initially assumed support for functioning and building projects only on computers with the MAC OS X system. Great success was achieved, and further work on the development of the engine and its tools was started. In 2009, the first version (2.5) of Unity compatible with Windows was launched, which turned out to be a great success [6]. The next version, Unity 3.0, was released in September 2010, with more tools that were previously available only in high-end studios appearing in the engine. The addition of such tools resulted in great interest in the company. One of the latest versions of the Unity 4.0 engine was released in 2012, with DirectX 11 support; Mecanim animation was added to it. Unity supports graphic resources and many file formats, from 3ds Max and Blender to Cinema 4D [7].

Unity engine version 2020.1.0a16 was used to create the “NeuroBall” computer game. The initial version selected for creating the game was 2019.2.18. However, due to problems with the use of an asset, it was changed to the newer version, where the problem related to importing the asset into the game world was already fixed. The Unity graphics engine has a very rich interface and many different functions. The program allows us to create scripts, for example, in C Sharp in the Visual Studio environment and many other languages and environments. The Unity interface consists of multiple windows, each with a different function, all of which interact with each other.

2.2. BCI Technology

The first research on BCIs began in the 1970s in Los Angeles at the University of California. The first publication on BCIs by Jacque was then published in Vidal with the title “Toward Direct Brain–Computer Communication”. In recent years, very intensive work has been done on BCI technology, which is why traditional cerebral readings using electroencephalography are now possible [8]. Being able to read brain waves, we can send them via the EEG to a computer or, for example, a smart wheelchair. Neurogaming, with its beginnings in the early 2000s, is one of the newer areas in the world of computing. One of the first games designed to help improve people’s mental processes was called “Brain Age Nintendo” released in 2005. Over time, many games using motion sensors appeared, for example, Microsoft for the Kinect controller or Wii Fit Nintendo; advanced training applications were also launched.

Electroencephalography (most often used to build BCIs) has many advantages. For example, it has the ability to work in various environments and is very easy to use (also independently at home). Compared to tomography or intracerebral electrodes, it has the lowest implementation cost ratio [9]. The EEG signal is read by means of special electrodes placed on the head, which have been previously soaked in saline or a special gel. The signals read from the electrodes are transferred to the amplifier and then, after amplification, sent to the computer. The data from the electrodes to the amplifier are sent via cables located in the body of the device. EEG signals are most often used to control the mouse, keyboard, or prostheses [10].

The Emotiv Epoc + NeuroHeadset is a wireless headset—an example of the practical implementation of BCI technology, thanks to which it is possible to record electroencephalographic waves from 14 channels. The headset is also equipped with two reference electrodes. This headset, unlike other devices that use EEGs, which, in order to function, often need large amounts of sticky gels, uses saline to wet the sensors. The device has two main sensors (reference sensors) that collect EEG motion data and axis data. All collected information is sent wirelessly via Bluetooth to the computer [11]. The Emotiv EPOC+ Neuroheadset is used to monitor brainwave activity in the bandwidth of 0.16–43 Hz and digital notch filters at 50 Hz. The EEG signals are sampled with 128 Hz. The electrodes in this electroencephalogram are fixed in the following positions: AF3, AF4, F3, F4, F7, F8, FC5, FC6, T7, T8, P7, P8, O1, O2, while the reference electrodes are located at P3 and P4. The headset shown in Figure 1 will be the device responsible for the player’s movement in the game “NeuroBall”. The main features of the Emotiv Epoc + NeuroHeadset are as follows: quick configuration that takes about 5 min, wireless, and motion sensors detecting head movements.

The Emotiv Epoc + NeuroHeadset was used in the project of the computer game “Neuro Ball” as a system that will allow us to move the player. The Laboratory of Neuroinformatics and Decision Systems of the Opole University of Technology is equipped with the Emotiv Epoc + NeuroHeadset, where the tool was tested. The game was tested on 10 students who volunteered to test the initial solution. All participants were instructed before the game started. All four levels of the game were completed by each player.

To start the Emotiv Epoc + NeuroHeadset, it must first be charged using the supplied USB cable, which is connected to the back of the device, and then it is plugged into the USB connector on the computer. The red LED on the body of the device means that the headset is charging, and the white one means that the device is fully charged. For the device to work, a special producer software (Emotiv Control Panel) must be installed. After charging the headset, we disconnect it, and then, using the box with sensors, soak the felt with saline. The soaked sensors are placed sequentially on the headset arms in a short twist motion. The headset, prepared in this way, can be turned on thanks to the button at the back of the device; then, we put it on the head with a movement from the top and gently push it on. The last element is connecting a Bluetooth receiver to the computer, thanks to which the Emotiv Epoc + NeuroHeadset will automatically connect to the computer.

2.3. Neurogaming

The topic of neurogaming was raised for the first time in San Francisco at the ‘NeuroGaming Conference and Expo’ organized by Zack Lynch on 2 May 2013. Neurogaming uses BCIs that take the player to a completely different level of gameplay because we do not need a keyboard or a mouse to control the objects in the game world [12]. The user communicates with the computer through the installed software and with a special device that uses the user’s brain waves, emotions or gestures to control the game [13,14]. Neurogaming can also be used to connect with many technologies, for example, virtual reality (VR) [15] or augmented reality (AR) [16], thanks to which the game will be supplemented with more new experiences. Such solutions enable game developers to better understand and control the emotions of players, which will allow us, in the future, to create individualised games and will take game development to a completely different level.

Through the use of BCI technology in neurogaming, apart from the entertainment of using a computer without the use of a mouse and keyboard, many other benefits are obtained as well. The use of such technology in specially designed games can help people with various health problems. Significant improvements have been seen in children suffering from hyperactivity by exercising the brain in a game that requires the concentration necessary to move objects in the game. Thanks to neurogaming, it will be possible to exercise and train a healthy brain, as well as treat various symptoms of its disorders and diagnose it in this area [10].

3. Results

3.1. Modeled Game World

The “NeuroBall” game was created in the Unity graphical environment in version 2020.1.0. From the beginning, the objective of the game was to create an interface that would encourage the youngest children to exercise their brains. Therefore, the game has been designed to be clear and pleasing to the eye. The main character, i.e., the ball, makes its journey through places surrounded by nature.

The first element of the game “Neuro Ball”, created in the world of Unity, was the floor, and, more specifically, a large flat surface on which we can place other objects and a player who will be able to move later on. The substrate also obtained a green texture to imitate grass (Figure 2a). The next step was creating further surroundings, i.e., mountains visible in the distance, in order to give the game more depth (Figure 2b). Each game must have its limits; therefore, a fence was created. In one part of the board, tall grass forms the borders, and, on the other side, a wooden fence made of boards (Figure 2c).

The next step was to add a variety of vegetation. There are many types of trees, flowers and shrubs on the board, which make it easier for the user to navigate the game world; it also improves the mood [17,18], thanks to which the use of the Emotiv Epoc + headset is more effective. The arrangement of the vegetation, however, had to be planned in advance, as some obstacles had to be placed in the game, and there had to be room for them (Figure 2d). The game world also has many rocks and stones of various shapes and sizes that further enrich the world with greater naturalness (Figure 2e). Due to the fact that the game has 4 levels, each of them had to be created separately. To create obstacles in subsequent levels, among others, wooden footbridges and wooden piles were used, from which the first two levels were created, i.e., a flat footbridge and a ramp (Figure 2f).

The creation of the third level, i.e., the “zigzag”, was carried out with the help of wooden boxes and a fence, from which the zigzag was created (Figure 2g). In the last level, there are a lot of large flat stones placed on the ground (Figure 2h). Collectable items have been created so that the user will want to move on to the next stages of the game and get points for them. In this case, they are coloured spinning crystals, and after we collect them, we can additionally hear a sound. Due to the fact that in the game, we can collect points, an inscription has also been added, which displays the number of points scored throughout the game. The introduction of scoring into the game was driven by the issue of a willingness to adopt a behavioural approach called reinforcement learning from Skinner, which promotes the use of rewards to assist the learners in progressing faster [19]. Each level must also be completed so that we can proceed to the next stage. Therefore, large, yellow stars have been placed on the board; after collecting them, the player moves to the next level.

When the creation of the game levels had been completed, the next step was to create the player himself. In this case, the object the user will move is a large red ball. It was chosen because everyone can imagine a rolling ball, which makes it much easier to work with the BCI. The control details are described in the next section. A script responsible for movement and colliders have also been added to the player’s object, i.e., the ball, thanks to which the ball can collect points and interact with other objects (Figure 2i). One of the last elements was adding lights to the world so that all elements can be clearly visible (Figure 2j). The last step was to create the game menu, in which the user has the option of using three buttons: start, volume setting and exit.

3.2. Control Software Configuration

The Emotiv Epoc Control Panel (Figure 3) is the main program in which the user can configure the headset. The program is divided into several modules. The first module is Headset Setup, where there is a schematic of all sensors. At the time of putting on the headset, there will be coloured points on the diagram to signal whether a given sensor is correctly placed. The green point means that the signal from the sensor is correct, and the red one means that it is not there at all. The module also has instructions on the correct use of the headset, and, in the upper part, there is the current status of the device, with information about the charge level of the headset and the quality of the wireless connection. Cognitive Suite is a module that uses a virtual 3D cube (Figure 4). The cube is animated and is used to detect cognitive functions. After selecting a specific training and the level of difficulty, the selected action is visualised in focus. On the left side of the module, there is an indicator that shows the level of the user’s focus.

EmoKey is a program that connects to the Emotiv Epoc Control Panel to access preconfigured actions; this is the final step in the configuration. Figure 5 presents the configuration of keyboard buttons used to move the player in the “NeuroBall” game, with actions configured in the Emotiv Epoc Control Panel. Each line is set to one button, and an action is assigned to it. EmoKey works constantly in the background, so we can start playing after putting on the headset. EmoKey affects the screen on which we are currently working, so if the game is on, the player’s control buttons are replaced with EmoKey actions. For example, an activated notebook will save the letters that have been assigned to the action.

For control of the ball in the game, EEG signals collected from the head surface of the subject are used with the Emotiv EPOC + NeuroHeadset, working in accordance with the 10–20 system of the International Federation for Clinical Neurophysiology IFCN. For this purpose, after correct device configuration in EPOC Control Panel, including using the Cognitive Suite tab, we use EmoKey. This is a program for creating macros, thanks to which we can assign buttons from the keyboard to functions such as push, pull, left, or right, previously set in the Cognitive Suite Emotiv EPOC Control Panel module (Figure 4). The time of the game is selected so that the person maintains proper concentration. It takes about 8 min to complete the level with full concentration. In order to carry out the process of controlling the ball in the game, a person must relax and then try to generate the states of brain activity so that they correspond to the previously coded actions in the Cognitive Suite. In the Cognitive Suite module, specific events are set and saved that are to be performed later in the game using the headset. On the left side (Figure 4), there is a 3D figure that allows us to visualise the set by the user actions; a bar above that shows the level of focus. On the right, the selected action is set, for example, push. After selecting the action, go to the training tab above, in which the user, in full concentration, tries to imagine how the object is pushing for a few seconds. After setting the action, the 3D figure should begin to recede. Adding more actions becomes more difficult and can make it difficult to move the subject because performing other actions and visualising requires a lot of practice and the ability to focus, which affects the training of the brain. The classifier used in this pilot study is built into the Cognitive Suite from Emotiv EPOC Control Panel tool used.

3.3. Numerical Results

Numerical data were presented to verify the impact of the game on the concentration level. First, an increasing concentration level was verified during the game (

Table 1. Numerical data of the increase in the average frequency of the EEG signal and the number of points scored within the concentration level.

Participant Number	Level	Average f of the EEG Signal in 1–3 Minutes of Playing [Hz]	Average f of the EEG Signal in 5–8 Minutes of Playing [Hz]	Number of Points Scored	Participant Number	Level	Average f of the EEG Signal in 1–3 Minutes of Playing [Hz]	Average f of the EEG Signal in 5–8 Minutes of Playing [Hz]	Number of Points Scored
1	L1	13	26	6	6	L1	15	30	7
	L2	25	30	7		L2	29	35	8
	L3	29	37	8		L3	30	37	9
	L4	35	40	8		L4	36	41	10
2	L1	18	32	7	7	L1	10	32	6
	L2	31	35	6		L2	23	37	6
	L3	32	38	8		L3	35	38	7
	L4	39	42	9		L4	38	42	8
3	L1	24	31	5	8	L1	22	34	5
	L2	21	34	6		L2	30	38	6
	L3	33	35	7		L3	30	35	7
	L4	35	43	8		L4	34	38	8
4	L1	21	37	6	9	L1	17	22	7
	L2	27	32	7		L2	21	26	7
	L3	31	36	8		L3	26	30	8
	L4	37	43	9		L4	32	35	8
5	L1	20	38	5	10	L1	19	30	7
	L2	28	39	7		L2	31	34	8
	L3	33	37	9		L3	31	38	10
	L4	31	41	10		L4	32	43	10

) for 10 participants. The data show that with the increase in levels (from 1 to 4), the concentration level identified in relation to the frequency of EEG waves increases in the case of most of the respondents. In addition, the number of points obtained by them increases, which is the result of higher concentration.

Table 2. Time to solve math problems before and after training with NeuroBall.

Participant Number	Time to Solve the Task before Training (Playing NeuroBall) [s]	Time to Solve the Task after Training (Playing NeuroBall) [s]	Participant Number	Time to Solve the Task before Training (Playing NeuroBall) [s]	Time to Solve the Task after Training (Playing NeuroBall) [s]
1	278	178	6	350	182
2	248	168	7	340	197
3	233	156	8	300	201
4	255	148	9	244	188
5	269	176	10	271	173

shows in what time the group of respondents managed to solve the mathematical task before and after concentration training through the game. The presented data show that a task of similar difficulty was solved faster after the concentration training carried out thanks to the game than before the game. All participants solved the same ordinary differential equations and others before and after the game, but with the same degree of difficulty. The group was also selected in terms of the results obtained in the field of mathematical analysis.

4. Discussion

To place a fence around the entire board so that the player will not fall outside of it was an important element in the creation of the “NeuroBall” game. The role of the fence is played by tall grass, a wooden fence and stones. The game world has various types of vegetation, which makes it easier for the user to pass through the levels, which affects the achievement of the appropriate alpha frequencies during relaxation between the activities that require concentration. At that time, high frequencies of electroencephalographic waves were observed during the research. At each level of the game, we can collect coloured crystals for which the player receives points, which additionally mobilises the player and, thus, stimulates the concentration process. The fourth and last level allows the user to move freely and use the directions in order to collect as many crystals as possible, which further increases the concentration effort and, thus, translates into active brain training. In the game “NeuroBall”, the player’s object is represented by a large red ball. The ball was chosen not accidentally but to make the process of steering and focusing attention the fastest possible and most effective.

When creating games, especially in the field of neurogaming, developers should pay special attention to light, which is a very important element in the game. Thanks to it, all elements are nicely lit and clearly visible, which affects the concentration process. The game “NeuroBall” uses directional light.

5. Conclusions

The aim of the work is to create a computer game project using neurogaming in correlation with BCI technology and the Unity graphics engine. This goal has been achieved. The computer game “NeuroBall” uses as the BCI the Emotiv Epoc + Neuro-Headset device. As shown by the tests, calibrating the device requires time and focus due to the fact that all recorded user behaviour will later be visible in the game. During the game, the focus and strenuous work of the brain are noticeably improved. The tests carried out on a few voluntary users showed that the game required their attention.

In the future, the game can be developed with various additional elements, including more levels of gameplay. In addition, it could be expanded with a multiplayer game, which would further enrich the user experience of BCI technology. The development of proprietary software for the classification of brain waves for the purpose of controlling an object in the game is also planned.