A Mixed Reality-Based Platform towards Human-Cyber-Physical Systems with IoT Wearable Device for Occupational Safety and Health Training

Abstract

Occupational safety and health (OSH) should be regarded as a crucial challenge that affects the public world widely. Work-related accidents and occupational illness contribute to considerable mortality and morbidity. As technology advances, mixed reality (MR) has gained popularity. To minimize occupational accidents occurring in the workplace and reduce human training time, an MR-based platform for OSH training combined with CPS and IoT technology is proposed in this paper. Multi-criteria decision-making (MCDM) and fuzzy-analytic hierarchy process (FAHP) were applied to evaluate and select suitable gloves. Only when the MR wearable devices are improved can a more powerful MR-based OSH training program be established. A higher immersive level of OSH training offers people a more realistic experience. They will better understand possible risks in workers’ future work, resulting in a lower occupational accident rate in the workplace.

1. Introduction

The Internet of Things (IoT), the Internet of Everything, or the Industrial Internet, acts as a global network of devices and machines that can interact [1]. IoT is considered one of the essential technology domains for the future and attracts vast attention from various industries [2]. The IoT endpoint market has grown enormously since the idea was published. However, robust interactions of human–to–device and device–to–device are achieved by the IoT applications [3]. There are numerous domains of IoT applications, such as healthcare, industries, vehicular communications, cloud computing, fog computing, edge computing, wireless sensor networks (WSNs), data mining, cellular networks, and many others [4]. In the future, there will be more and more devices available to interact with the system of IoT [3]. Companies have developed IoT solutions for facility maintenance and cold chain management applications. Taking facility maintenance as an example, with the help of multiple sensors and IoT systems, functions such as motion detection in the restricted area and water detection in the pump room can be achieved for the ease of facility management [5,6,7,8,9].

In the age of IoT, progress allows the number of advanced IoT applications such as smart healthcare systems, intelligent transport systems, smart energy systems, and smart buildings [10]. Sensors are the cornerstone of IoT deployment [11,12,13,14,15,16]. They enable IoT by collecting valuable data for better decision-making, especially for health and safety. The capability of sensors is aimed at securing the dangerous working environment, preventing injuries, and predicting risks. Intending to achieve the above targets, IoT sensors are helpful in occupational safety and health training [17,18,19,20]. Occupational safety and health (OSH) is multidisciplinary and concerned with people’s safety, health, and welfare issues at work. It keeps a watchful eye on workplace risks, including physical, chemical, biological, and psychological hazards and accidents. The primary target of OSH is to make improvements in offering a safer and healthier working environment, which means lessening the number of occupational injuries and deaths, musculoskeletal disorders, harmful occupational exposures, sick leave, and worker complaints [21].

IoT cannot be divorced from cyber–physical systems (CPS) to maximize advancement opportunities. They are both sophisticated platforms that aim to enhance technological reliability and expose fields of unexploited possibility [22,23,24,25]. CPS is a complex, heterogeneous distributed system of collaborating computational factors to control physical objects [26,27,28,29,30,31,32]. The physical side is connected with the sensors, actuators, and hardware and transmitted to the network and cyber layers. CPS aims to detect and realize the variations in the natural environment, analyze the influence caused by the changes to the operation and make decisions to answer the corresponding changes through dispatching commands to control physical devices in the system. This feature enables remote physical environment control [33]. Moreover, IoT is a technology that connects different independent equipment into one communicational platform for data monitoring and processing [3,17,19,20,34]. CPS and IoT are involved in integrating digital competencies such as computational abilities and network connectivity with physical objects and systems to promote performance [35,36,37,38,39]. Examples include intelligent vehicles, smart grids, cutting-edge manufacturing systems, and wearable medical devices [7,8,40,41,42,43,44]. Wireless communication plays a leading role in the smart world. CPS and IoT are standard techniques directly interacting with innumerable human daily activities [33]. CPS’s cooperation with IoT can produce remarkable progress in various industries. Fully grasping the distinctions and the relationship between IoT and CPS is of the utmost importance [33,45,46,47,48,49,50,51].

1.1. Wearable Hand Device

Conventional gesture recognition is conducted through wearable hand devices like data gloves. It is a wearable sensor input device of human–computer interaction applied for hand motion capture. When users wear the gloves and perform some hand gestures or sign language, the data gloves can grasp the critical information of the hand gesture, such as position, orientation, and configuration, and transfer them to the computer. Then, the finger and hand motion is accurately transformed into real-time virtual information. A feed-forward neural network for gesture recognition can train through adopting the collected data. The sensors of data gloves play a vital role in acquiring gesture-related information. Without sensors in data gloves, the hand motions will not be interpreted into digital data and transmitted to the host computer for further analysis. The performance of sensors may affect the degree of precision of the selected information, which directly influences hand gesture recognition accuracy and completeness. The recognition workflow of data glove is indicated by the following procedures. Initially, users wear the data gloves and display different gestures in the data acquisition phase. The embedded sensors can sense the flexing point of each joint of fingers and deliver the collected data to the microcontroller to capture gestures. Those input signals will then be processed and become proper signals for feature extraction. After extracting the data, the processed data are transmitted to a gesture recognition and hand tracking system for training the algorithm, getting the recognition rate, or others. Lastly, the hand gestures are realized and translated into corresponding meanings. There are pros and cons of wearable hand devices. The data gloves can measure the parameters of the hand and finger directly. They are easy to use for acquiring helpful information with fast processing speed. However, using the data gloves for hand gesture recognition may obtain several limitations. The data gloves are time consuming with regard to dynamic gesture recognition. The recognition accuracy is highly dependent on the data collection process, which cannot obtain a stable recognition performance. If the data collection procedure is conducted well, the recognition accuracy will be higher and vice versa. Furthermore, this approach suffers from slow enforcement, which may not be appropriate for real-time interactions. In addition, data gloves are immersive devices that may result in an inconvenient operation and be less comfortable worn by the users. Data gloves are unsuitable for long-distance control and MR-based applications [52,53,54].

1.2. Mixed Reality

Combining the IoT system with MR technology, the environmental parameters of the site intended for simulation can be caught accurately, which is beneficial to the development of an immersive situation for OSH training. Another advantage is that the operator’s condition can be monitored for data analysis and a better case study. With the growing trend and popularity of mixed reality (MR), augmented reality (AR), and virtual reality (VR), the confusion about the divergence between these three innovative technologies is rising correspondingly. Although all of these immersive techniques have common properties, they are different. To better comprehend MR, AR, and VR and their interrelationship, the term the continuum is used. This is bounded by an authentic environment that means the real world and an entirely virtual environment such as VR with augmented reality (AR) and augmented virtuality (AV). MR is defined as everything between real and virtual environments. AR is merely a subclass of MR, and VR cannot be counted as a portion of MR. MR is the acronym for mixed reality. Simply speaking, MR is an integration of AR and VR [55,56]. It combines digital and real worlds to unblock the linkage between human, computer, and environment interaction [57,58,59,60,61,62,63]. Users are enabled to communicate with digital items placed in the physical world in real-time. Moreover, the virtual objects will respond to users as real things [15,55,64]. MR should be counted as the latest immersive technology among the three. For example, the information about a surgical room is embedded in the actual scene. By adopting MR technology, users can interact with the data using a surgical device. Conducting simulated surgeries repeatedly allows the surgeon to prepare backup solutions when minimizing the chance of accidents occurring and encountering related accidents. MR acts as a new tool that can provide an opportunity to expand educational and training methodologies. As an immature technology in the early stage, MR has some limitations. To experience MR, equipment, such as an MR headset for a credible and three-dimensional experience is needed. However, head-mounted display devices, such as the headset, HoloLens, and Google Glasses, restrain or even stop the head movement of users. Thus, devoting more efforts to MR development and upgrading the functionalities of wearable devices are the prerequisites for improving the immersive user experience.

1.3. Occupational Safety and Health Training

In Hong Kong, the Labour Department is responsible for OSH issues [65]. The OSH ordinance covers most of the workplaces that employees work in, such as factories, construction sites, laboratories, offices, educational institutions, shopping arcades, and catering establishments. Nevertheless, there are some exceptions. The Commissioner for Labour is authorized to enforce the ordinance by improving the notices and suspending notices to avoid imminent hazards to employees during workplace activity [65]. Under the legal framework, OSH in Hong Kong is constructed to facilitate the all-inclusive establishment of OSH and preserve the OSH of every society party. However, there are still some imperfections in real-life operations. To reduce the operation cost, some organizations may escape from OSH ordinances. Moreover, the current regulatory framework for OSH mainly focuses on industrial corporations, while there is no relevant constraint to other kinds of institutions in Hong Kong nowadays. With the progress of technology and society, more and more organizations are applying new technologies, which brings out new OSH problems, including working under pressure, overtime, work-related hazards brought by 3C products, and other vocational risks. To improve OSH fields that are imperfect or undiscovered and save workers at risk for injuries, a more organized framework of OSH management by laws is necessary. Around the globe, good OSH services have merely existed in 5% to 10% of employees’ workplaces in developing countries and 20% to 50% of those from industrialized nations [66]. According to the International Labor Organization, nearly 1.9 million people suffer from occupational diseases, and 2.3 million employees die annually from work perils [67]. Construction is one of the industries with the most severe accident or injury rates. A study revealed that exposure is crucial for construction workers’ injuries [68]. OSH ought to have high precedence on the international agenda, while improving OSH infrastructures and systematic preventive approaches in industrializing countries is prolonged. Under the circumstance, education and training within the OSH in advance of employment might be vital. Basic knowledge and compliance with rules are two essential elements that OSH training focuses on it. To prevent risks and establish a culture of prevention, training in safety rules is a lever to be activated.

1.4. Problem Description and Objectives

In addition, preliminary research about MR-based training should be counted as another problem. VR was more commonly used than MR for safety training, and training was prominent in the maintenance of workplace safety. Due to the lack of research in MR-based training, there might not be a significant chance to incorporate IoT and CPS in the training applications. Enriching MR content using IoT requires new architectures to handle the complexity of MR integration within the IoT platform. The functionality of CPS lies in conducting cloud-based data analysis in the cyber layer [69,70,71]. An MR technology-based OSH training enables trainees to explore hazardous situations without exposure to real-life threats. Thus, a lack of research in MR-based training can be a significant obstacle to uplifting the training quality and effectiveness. Referring to the study by the Bureau of Economic Analysis [72], the estimated expenditure on work-related accidents and occupational illnesses is around USD 200–550 billion. The data disclosed that OSH is a fundamental challenge in the workplace worldwide. Hence, this research expects to achieve two objectives. The first objective is to minimize errors or accidents in the working environment to protect workers from exposure to health and safety risks. The second objective is to reduce the training time to ensure work sufficiency by using new technical apparatus. Adopting advanced technologies is of utmost importance to ensure the occupational accident rate can be declined. To overcome insufficient research in MR-based OSH training and the absence of MR application with CPS and IoT for OSH training, employing CPS and IoT techniques to establish a better-quality MR-based platform for OSH training should be considered as one of the suitable solutions. Therefore, the main research questions that need to be addressed are as follows:

• How could an MR-based environment be assisted for OSH training?
• How could HCPS be adopted for OSH training under an MR-based environment?

2. Theoretical Background and Related Works

This section reviews state-of-the-art research in MR-based training platforms and the use of IoT and CPS-based system architecture for OSH training.

2.1. IoT-Based Occupational Safety and Health

IoT has proved to be one of the leading exponents in communication and attracted considerable research attention in the twenty-first century. It allows vast amounts of items, including sensors, electric apparatus, or vehicles, to be connected to the Internet, offering helpful information, data, and resources [17,19,20,34,73]. IoT is not merely knowledge and communication technology but also technology in the safety and health sector. It plays a crucial role in healthcare and ambient monitoring applications. For instance, wireless sensors adopted in different spots observe surrounding environments, and sensors of wearables can be attached to workers’ bodies to measure physiological conditions. The collected information will then be transmitted to the cloud infrastructure and delivered to target receivers [74]. In addition, construction is a challenging field from the viewpoint of OSH. The safety and health of workers are momentous considerations on a construction site. Mishaps of all kinds at the construction site will cause occupational injuries and illness to workers, hence decreasing their working effectiveness as well as changing their life permanently [75]. The emergence of IoT and its concomitant technologies such as wearables have raised interest in the OSH of health and safety monitoring applications and building work [74]. As the health status of workers and environmental parameters of the construction site are collected, proactive prevention can be taken after data analysis to lessen the occupational accident rate in the construction industry [41,43,76,77,78,79,80]. In short, IoT provides instant and consistent support in monitoring various human and environmental metrics. Therefore, IoT-based OSH can improve the safety and productivity of the industry and minimize workers’ health impacts [18,80,81,82,83].

2.2. CPS-Based Occupational Safety and Health

The cyber–physical system (CPS) is also known as an intelligent ICT system. This system consists of interconnected, interdependent, cooperative, and autonomous features. It provides computing and communication capabilities to monitor and control physical objects in various applications. The concept and functionalities of CPS can be applied in the workplace to enhance the OSH [18,20,42,82,84,85,86,87]. Generally, the notion of CPS can be demonstrated as an imagination infrastructure. It is comprised of two layers, including the physical and cyber layers [88,89,90,91,92]. The physical layer consists of physical entities located in physical 3D space to implement particular jobs and communicate with each other physically. The entities might be composed of sensors and actuators or equipped with proper sensors and actuators. Spatially distributed telecommunication nodes and computing networks are part of a cyber-layer. This is connected to sensors or actuators and immersed in the natural environment [93]. As mentioned above, CPS contains various sensors and actuators employed to monitor abundant parameters of workers’ health and the surrounding environment. These assume that the data processing units, transmitters, and receivers are installed in sensors and actuators. Hence, the workers’ health states and corresponding working environment can be monitored and controlled by data processing to guarantee the OSH [88,89,90,91,92]. Utilizing real-time monitoring, the health status of laborers can be observed by measuring primary physiological coefficients like heart rate, body temperature, breathing rate, and work comforts like work posture, underclothing temperature, and humidity. Moreover, workplace hazards such as exposure to poisonous chemical matters, noise, and optical radiation can be detected. After analysis, these collected data will warn workers about the emergence of risky situations. Furthermore, the protective systems will activate after surpassing a high-risk threshold value. Therefore, CPS should be considered a tool to improve OSH. Hence, it should be adequate to prevent workers from being exposed to the risks of health hazards and prone to disease. Furthermore, healthcare and medicine applications are on a medium scale. The adoption of CPS in this domain helps to monitor the workers’ health conditions and take necessary responses to ensure their safety and health.

2.3. Types of Hand Wearable Device

The concept of wearable intelligence has been broadly explored in the past decades. Numerous emerging technologies offer convenience to daily human life, especially for old and disabled people [94]. The adoption of hand wearable devices is not new to our world. Hand data gloves are applied in various study domains, including medical surgery, game, virtual reality, and shopping applications [95]. Data gloves achieved a high accuracy rate suitable for medical surgery. In gaming, gestures are the most interactive module for game control. Even iPods, iPhones, or iPads use gestures on mobile video game platforms. Gestures must be recognized first, and hence the data glove is employed. The data glove is designed to replace static and fixed keyboards and mice to promote a sense of immersion. It permits users to interact with the computer more authentically and naturally [96]. In the aspect of rehabilitation, the hand wearable device is of utmost importance. A research project revealed that the incidence of strokes has risen twofold in low-income and middle-income countries in the last three decades [97]. Stroke is the primary cause of physical disability. After having a stroke, complications such as upper limb hemiparesis are common [98]. Stroke patients can use smart gloves outside of the therapy clinics without supervision by therapists or healthcare providers. Researchers have explored adopting state-of-the-art techniques to enable hearing and speech-impaired people to communicate and build ties with others. A sensory glove can collect and record information for dynamic and static signs to facilitate the accuracy of sign language or gesture recognition in real-time and interpret the meaning into words [95]. Therefore, the invention of wearable hand devices such as data gloves provides a more natural and resultful approach to communication, particularly for people with special needs. Furthermore, wearable hand device usage is being explored continuously to bring humankind convenience.

Hand Wearable Devices Related to Safety and Health

Nowadays, safety and well-being at work should be recognized as urgent affairs for numerous industries. Wearable devices are one of the most hopeful solutions to eliminate or mitigate the hazard of mental and physical accidents in the working environment. Organizations are broadly employing wearables to perform different tasks to improve workplace safety [67]. Hand wearable devices are one practical example of wearables for OSH monitoring. In the high-temperature workplace, protective gloves can release heat-related warnings and alert messages to support and protect firefighters or workers [93]. The embedded wireless system in protective gloves consists of an analog temperature sensor for observing temperature on the opisthenar of a worker, a barometer for detecting the changes of atmospheric pressure, and a thermocouple for measuring contact heat as well as providing haptic feedback in the event of risky situations by miniature vibration motors that are embedded in the flexible part of the glove. Thus, this powerful hand wearable device has emerged for substituting occupational risks, injuries, and diseases [99]. Latent hazards such as workers located in a dangerous area, the worker being under high physical demands or fatigue, and the danger of worker exposure to musculoskeletal disorders can be effectively avoided or lessened by inferring from the collected information [100]. Therefore, the advanced sensors and technologies offer plenty of chances for real-time monitoring to maintain adequate workplace safety and health management and identify OSH risks through timely feedback. Since OSH is a global priority, the potential of hand wearable technology will enable a new path toward maintaining OSH. Adopting hand wearable electrons in the workplace is being promoted to uplift employees’ health and well-being. In conclusion, occupational hazards can be identified, reduced, eliminated, and controlled.

2.4. Research Gap

In IoT-based OSH, there are limited studies investigating the potential advantages of IoT-based professional safety and health solutions to society. More comprehensive knowledge-sharing research results are necessary to demonstrate the societal and economic benefits of utilizing IoT technologies in the working environment to guarantee OSH [75]. Furthermore, privacy and security issues cannot be separated from IoT-related techniques. Insufficient information about the data collection processes in the worksite forms results in distrust of the technology [67]. Due to data security being of the utmost importance for personal health information, the IoT network system should collect more subjects to validate and uplift the reliability and privacy level [74]. In addition, researching user-centered approaches in the occupational IoT for retaining OSH would prove significant. In CPS-related OSH, context-awareness has been used in applications for running CPS to guarantee workers’ safety in the working environment. However, only a few studies have been published for this specific domain. For instance, more than 300 ICT solutions related to safety in the systematic inventory can be applied in the smart working environment; only a few of them are related to context awareness [93]. The result reveals that the development of CPS for OSH as an essential research field is still lacking. As CPS is a broad research area, further interdisciplinary research and revolutionary activities must offer bases in CPS’s hardware and software components design. As a result, the main goal of ensuring OSH for workers can be achieved. In the domain of hand wearable device that is relevant to OSH, only a few worksite hand wearables have undergone rigorous field studies and comprehensive verification against standards. With insufficient peer-reviewed information, it could be challenging to confirm the efficacy and safety of hand wearable devices in the market. Moreover, very few studies have investigated the social resistance to adopting hand wearable devices. The degree of acceptability is more crucial than the related functions and benefits in modern times [67]. Consequently, exploring the appropriate methods to boost the acceptance of hand wearable technologies at work is a prerequisite for preserving OSH and getting rid of the research gap.

3. System Architecture and Methodology

3.1. Human–Cyber–Physical System

A human–cyber–physical system (HCPS) should be considered a natural extension of CPS. ICT supports human communication and collaboration with cyber and physical systems [101]. The HCPS is among the most popular aspects of computing science and technology, communication, control engineering, and ICT application communities. The HCPS connects humans, material processes, social medias, and cybernetics to be a comprehensive system to distinguish intelligence communication, varied integration, and grand design [102]. According to Liu and Wang [101], HCPSs are established for using ad-hoc processes, especially in infrastructure establishment. Significant developmental fields are essential to the state economy and human well-being. Due to the possible enabling competence in digitizing human life and social and economic activities, most industrial countries and territories have developed tactics beyond 2020, including the Made in China 2025, German (European Union) Industry 4.0, and USA Industry Internet.

According to Zhou, Zhou, Wang, and Zang. (2019), the HCPS is related to human, physical, and cyber parts to realise a better information flow from physical to virtual. Human responsibilities and roles are dramatically changed in the industry caused of evolutionary fabricating techniques [101]. In the case of the HCPS, human and physical systems gain assistance from digitalizing and focusing on valued jobs and enlarging production by adopting machines or robots [73]. Moreover, the power of human resources and creativity can be freed up due to a progressively intelligent physical system that makes humans achieve the targets in an easier way. In the domain of intelligent fabricating, technical mechanisms, and the approach to constructing the technological architecture can be disclosed by HCPSs. To conclude, designing, constructing, and employing HCPSs in different cases and at diverse levels are the principles of intelligent manufacturing with safety training. With the rapid establishment of improved actuation, sensing, embedded computing, and AI, a harmonious human–machine–intelligence collaboration paradigm can be created by a HCPS. Hence, the HCPS becomes applicable in wider domains and more sophisticated health care and industrial circumstances [102].

The HCPS is proposed majorly for the OSH training to reduce the overall accidences and errors that appear during the actual working environment. The purpose of presenting a human-related CPS under the MR-based environment is to combine the traditional cyber-layer architecture with the MR environment for a simulation that could be capable of data analysis and forecasting. The human element is involved because the architecture requires human involvement during the physical layer, assisted with IoT devices. Therefore, the HCPS architecture is proposed with the MR-based environment for the OSH training. Figure 1 shows the conceptual diagram of the Human–Cyber–Physical System under the mixed reality-based training platform, while Figure 2 shows the system level of the human–cyber–physical system in OSH Training. The MR-based training platform is majorly considered for the digital part of sensor control, integration, perception, motion risks, and to provide discussion on the human. The aim is to digitalize the OSH training under an MR-based virtual environment and represent the models, information, and knowledge within the cyber layers. This system could combine the physical parts of the human operation, gloves for simulating the actual operation, and the actual working environment to further nearly-real-time synchronization from physical to cyber layers. The framework is proposed to consider human involvement within the traditional CPS. Under the system level, the proposed framework could further combine the information for forecasting and prediction. A closed-loop architecture is proposed with different stakeholders in OSH training.

3.2. Wearable Hand Device

To construct a wearable device that can fit most users, some ergonomic analyses have been carried out in the planning stage. First, the average dimension of the adult hand is obtained, shown in

Table 1. The average dimension of the adult hands.

Gender	Average Length	Average Breadth
Male	18.9 cm	8.4 cm
Female	17.2 cm	7.4 cm

. There is a slight variation in the measurement between the hand sizes of adult males and females. The construction material of gloves is determined in the next step. It would be made of elastic material to adopt the variation. The sensing device is based on

Table 2. Sensing device analysis.

Sensor Type	Usage	Relationship with Project
Temperature sensors	Detect a person’s skin temperature [103]	Track trainer’s skin temperature
Position sensors	Detect linear position. Detect rotary position. Detect angular position.	Track the movement of fingers, such as bending
Motion sensors [104]	Detect gesture or hand motion	Track trainer’s gesture during work (formal gesture or not)
Force sensors [104]	Detect the level of force used by the trainer’s hands	Track trainer’s applied force to the glove
Accelerometer with gyroscope [105]	Detect linear movement and orientation of the gloves	Track the position of the whole hand

. In order to fulfill the requirements concluded in the ergonomic analysis, two kinds of sensors are chosen. They are accelerometers with gyroscopes and flex sensors. The accelerometer measures linear acceleration in mV/g along one or several axes, while the gyroscope measures angular velocity in mV/deg/s. Through inputting the data from the accelerometer and gyroscope into the set location in the MR-based training platform, the virtual hands’ moving direction and destination can be fully represented by virtual hands. Figure 3 shows the proposed wearable hand device. A coin-shaped vibrator on each fingertip stimulates the feedback of interacting with different objects while using the gloves. To make the user experience closer to reality, a wide range of vibration strength and lasting time could be set. A clicker is located on the left side of the second joint of the index finger. It is used to act as a confirm key.

Before the construction of the glove, circuitry should be tested to ensure that there are no potential connection mistakes and component failures. If the step is not implemented before the construction of the glove, it will be difficult to distinguish between a connection mistake and a component failure, resulting in a waste of time and effort. The ESP32 datasheet provides information about the function of each pin. Pin 3V3, IO26, and GND are used for the button. Pin 3V3, GND, IO22, and IO21 are used for the MPU. Pin 3V3, IO2, and GND are used for the flex sensor. Pin IO4, 3V3, and GND are used for the heartbeat sensor. A flex sensor detects the movement of fingers by changing the numerical value of resistance. The sensor has a resistance of around 7000 to 13,000 ohms when in a flat situation. During bending fingers, the resistance value would increase—at least two times of flat resistance value when a 180-degree pinch bend is obtained. The proportion of the bend level and the resistance value can be found through calculation. Then, the fingers’ level can be estimated by analyzing the reading shown on the Arduino platform. A flex sensor is located above each of the fingers. An accelerometer with gyroscope and microcontroller is placed in the gloves’ middle. All the flex sensors and accelerometers with a gyroscope are connected to the microcontroller by wire. To achieve the target of a wireless wearable, a battery box placed on the first half of the forearm is connected to the microcontroller to act as a power source. Still, the range of the flex sensor is minimal. Having a high starting value, the flex sensor can reach the peak of the range very quickly, which is less than ideal for measuring the bend of the finger. It may be impossible to distinguish between a 90-degree bend and a 100-degree bend if the flex sensor reaches the limit while a slight bend is applied. Therefore, a voltage divider is a simple circuit that divides a large voltage into a smaller one, where V_in indicates the 3V3 pin, R1 is the flex sensor, V_out is the IO pin, and R2 is a 50 k ohm resistor. With the implementation of the voltage divider, the starting value of the flex sensor is decreased, and the bend applied to the sensor can be distinguished easily.

3.3. Multi-Criteria Decision Making (MCDM)

Three kinds of gloves can be easily bought at a relatively low cost. For these reasons, they have the highest priority in consideration. They are labor gloves, slip-proof gloves, and nylon gloves. A multi-criteria decision-making (MCDM) analysis has been built for evaluation shown in Figure 4. The goal, criteria and alternatives are also shown in Figure 4. A survey was conducted to obtain the opinion and perception of the glove’s selection assessment. One hundred questionnaires were distributed to the participants through Google Forms in 2021. All the questionnaires are received from undergraduate students in Hong Kong.

Table 3. Multi-criteria consideration.

	Cost	Comfortability	Processability	Flexibility
Cost	1	5	4	7
Comfortability	$\frac{1}{5}$	1	$\frac{1}{2}$	3
Processability	$\frac{1}{4}$	2	1	3
Flexibility	$\frac{1}{7}$	$\frac{1}{3}$	$\frac{1}{3}$	1

shows the summarized results.

Fuzzy Analytic Hierarchy Process (FAHP)

The fuzzy-analytic hierarchy process (FAHP), which would be used to get rid of the weight of criteria, is shown in

Table 4. Attribute of FAHP.

	1	2	3	4
Attribute or criteria	Cost	Comfortability	Processability	Flexibility

[106,107,108,109].

The most critical step in AHP is creating the pairwise comparison matrix. This pairwise comparison matrix is created with the help of a scale of relative importance. The values in the scale of relative importance are quiz numeric values. These values can be converted to fuzzy numbers shown in Figure 5 and Figure 6 [110,111,112].

Fuzzification is used in the fuzzy system. It can cause covert linguistic gum in the membership function. As the shape of the membership function is triangular, it is known as a triangular membership function.

$${\mu }_{Ã}(x)=\tilde{A}=(1, 2, 3)$$

(1)

µ_Ã(x) is the fuzzy value. (1, 2, 3) are fuzzy numbers associated with the membership function. These fuzzy numbers are the triangle’s lower, middle, and upper ends on the x-axis, shown in

Table 5. The fuzzy scale of relative importance [110,111,112].

	Numeric Value	Fuzzy Number
Equal	1	(1,1,1)
Moderate	3	(2,3,4)
Strong	5	(4,5,6)
Very strong	7	(6,7,8)
Extremely strong	9	(9,9,9)
Intermediate values	2	(1,2,3)
	4	(3,4,5)
	6	(5,6,7)
	8	(7,8,9)

.

On the scale of relative importance, the quiz numbers such as 1,3,5,7,9 are spirited with fuzzy numbers. It seems that assigning a single number to any thumb is not justified. Moderate has the given number (2, 3, 4), which are the triangle’s lower, middle, and upper points. The corresponding triangle is the membership function for the moderate. The intermediate membership functions are shown in the red lines.

Table 6. Pairwise comparison matrix.

	Price or Cost	Storage Space	Camera	Looks
Cost	1	5	4	7
Comfortability	$\frac{1}{5}$	1	$\frac{1}{2}$	3
Processability	$\frac{1}{4}$	2	1	3
Flexibility	$\frac{1}{7}$	$\frac{1}{3}$	$\frac{1}{3}$	1
	Price or cost	Storage Space	Camera	Looks
Cost	(1,1,1)	(4,5,6)	(3,4,5)	(6,7,8)
Comfortability	$\frac{1}{5}$	(1,1,1)	$\frac{1}{2}$	(2,3,4)
Processability	$\frac{1}{4}$	(1,2,3)	(1,1,1)	(2,3,4)
Flexibility	$\frac{1}{7}$	$\frac{1}{3}$	$\frac{1}{3}$	(1,1,1)

shows the pairwise comparison matrix.

Referring to the fuzzy scale of relative importance, the corresponding fuzzy number can replace the numeric value, but the fractional values are not converted into a fuzzy number.

$${Ã}^{-1}={(l, m, u)}^{-1}=(\frac{1}{u}, \frac{1}{m}, \frac{1}{l})$$

(2)

Using the above equation, the fractions can be converted to fuzzy numbers.

Table 7. Fuzzified pairwise comparison matrix.

	Price or Cost	Storage Space	Camera	Looks	The Fuzzy Geometric Mean Value $\tilde{\mathit{r}}$	Fuzzy Weights $\tilde{\mathit{w}}$
Cost	(1,1,1)	(4,5,6)	(3,4,5)	(6,7,8)	(2.91, 3.44, 3.94)	(0.428, 0.610, 0.859)
Comfortability	($\frac{1}{6 }$$, \frac{1}{5}$$, \frac{1}{4}$)	(1,1,1)	($\frac{1}{3 }$$, \frac{1}{2}$$, \frac{1}{1}$)	(2,3,4)	(0.58, 0.74, 1)	(0.085, 0.131, 0.218)
Processability	($\frac{1}{5 }$$, \frac{1}{4}$$, \frac{1}{3}$)	(1,2,3)	(1,1,1)	(2,3,4)	(0.80, 1.11, 1.41)	(0.117, 0.196, 0.309)
Flexibility	($\frac{1}{8 }$$, \frac{1}{7}$$, \frac{1}{6}$)	($\frac{1}{4 }$$, \frac{1}{3}$$, \frac{1}{2}$)	($\frac{1}{4 }$$, \frac{1}{3}$$, \frac{1}{2}$)	(1,1,1)	(0.30, 0.35, 0.45)	(0.044, 0.063, 0.099)

shows the fuzzified pairwise comparison matrix.

The geometric mean is used to calculate the weights. The fuzzy geometric mean value $\tilde{r}$_i can be calculated by the equation as shown below [106,107,108,109,110,111,112].

$${Ã}_1\times {Ã}_2=(l_1, m_1, u_1)\times (l_2, m_2, u_2)=(l_1\times l_2, m_1\times m_2, u_1\times u_2)$$

(3)

Here is the equation for adding two fuzzy values to get the summation:

$${Ã}_1\times {Ã}_2=(I_1, m_1, u_1)\times (I_2, m_2, u_2)=(I_1\times I_2, m_1\times m_2, u_1\times u_2)$$

(4)

Hence, fuzzy weights can be calculated.

$${\tilde{w}}_{\mathrm{i}}={\tilde{r}}_{\mathrm{i}}\times {({\tilde{r}}_1+{\tilde{r}}_2+\cdots +{\tilde{r}}_{\mathrm{n}})}^{-1}$$

(5)

In order to get the weights w_i, the center of area equation should be adopted: W_i = ($\frac{l+m+u}{3}$). The weights could be used as the requirement for further calculation. The total of the criteria weight is 1.058, which is not acceptable. Only when the normalized weights are equal to one can the normalized weights be applied for further calculation.

Table 8. Fuzzified pairwise comparison matrix.

	Fuzzy Weights $\tilde{\mathit{w}}$	Weights wi
Cost	(0.428, 0.610, 0.859)	0.633
Comfortability	(0.085, 0.131, 0.218)	0.145
Processability	(0.117, 0.196, 0.309)	0.207
Flexibility	(0.044, 0.063, 0.099)	0.068
	Weights wi	Normalized weights
Cost	0.633	$\frac{0.633}{1.058}$ = 0.601
Comfortability	0.145	$\frac{0.145}{1.058}$ = 0.138
Processability	0.207	$\frac{0.207}{1.058}$ = 0.197
Flexibility	0.068	$\frac{0.068}{1.058}$ = 0.065
Total	0.633 + 0.145 + 0.207 + 0.068 = 1.058	0.601 + 0.138 + 0.197 + 0.065 = 1

shows the fuzzufied pairwise comparison matrix and

Table 9. Eleven-point spherical fuzzy linguistic term scale [110,111,112].

Linguistic Terms	Spherical Fuzzy Number
Extremely low	[0.045, 0.955 0.045]
Very low	[0.135, 0.865 0.135]
Low	[0.255, 0.745 0.255]
Fair	[0.335, 0.665 0.335]
Medium	[0.410, 0.590 0.410]
Good	[0.500, 0.500 0.500]
Very good	[0.590, 0.410 0.410]
High	[0.665, 0.335 0.335]
Very high	[0.745, 0.255 0.255]
Exceptionally high	[0.865, 0.135 0.135]
Excellent	[0.955, 0.045 0.045]

shows the eleven-point spherical fuzzy linguistic term scale.

4. Results and Discussion

To illustrate our results, the construction industry has been chosen to be the target industry, and strip, trip, or fall-related accidents are the focus. In addition, the primary product of this project will be a glove that interacts with an MR device on hand movement. A reinforcement fixing at height MR training for construction site workers is suggested in order to combine these two elements.

4.1. Programming

The program follows a simple structure of four steps shown in Figure 7. The first step is to summon the serial monitor; the second one is to set up the pin mode, the third one is to read the data, which could either be digital or analog, and the last one is to print out the data on the serial monitor.

4.2. Experiments

Four experimental results will be demonstrated in this section: acceleration and coordinate, heartbeat, finger bending, and confirm button. When the glove moves in and out, the reading in acceleration X changes rapidly, indicating that coordinates in and out are the x-axis of the glove. When the glove moves left and right, the reading in acceleration Y changes quickly, meaning that the coordinates left and right are the y-axis of the glove. When the glove moves up and down, the reading in acceleration Z changes rapidly, indicating that coordinates up and down are the z-axis of the glove. Secondly, the reading of the heartbeat sensor was around 3400~4095, which is not the exact number of heartbeats, such as 60~100 beats (average heart rate). The interpretation of the heartbeat sensor reading failed due to limitation of knowledge. The heartbeat sensor is currently useless, which should be improved in future work. Thirdly, for the finger bending, the raw reading of the flex sensor is around 0~4000 and interpreted into angles (0~180 degrees) displayed in the video. The interpretation is made according to the following steps:

○

Wear the glove and perform three finger positions: flat, 90-degree bends, and 180-degree bends. Record the raw readings of these three positions.

○

Analyze the raw readings and interpret them into angles logically.

○

Develop a program to perform the interpretation using the logic developed.

During step 1, the raw readings of the flex sensor are found in

Table 10. Finger angle and raw reading boundary.

Finger Angle (FA)	Raw Reading Boundary
0-degree	less than 1500
90-degree	less than 3500
180-degree	less than 4000

. Therefore, if the primary reading is less than 1500, FA is 0; otherwise, if the raw reading is less than 3500, FA is 90; otherwise it is 180. The intermediate angle can be calculated roughly by raw reading minus the previous boundary (1500 or 3500) times the boundary difference over 90 degrees. For example, if the raw reading is 2450, the angle will be (2450 − 1500)/(3500 − 1500/90) = 42.75-degree.

Table 11. The pseudo-code of the finger bending and flex loop.

1	//Flex Loop
2	int Flex = analogRead(Flex_Pin);
3	//Serial.print(“Flex: “);
4	//Serial.println(Flex);
5	int FA = 0;
6	if (Flex ≤ 1500){
7	FA = 0;}
8	else if (Flex ≤ 3500){
9	FA = (Flex − 1500)/(2000/90);}
10	else if (Flex ≤ 4000){
11	FA = (Flex − 3500)/(500/90);}
12	else{
13	FA = 180;}
14	Serial.print(“Flex: ”)
15	Serial.print(FA);
16	Serial.println(“ degree”);

shows the pseudo-code of the finger bending and flex loop. Fourthly, the button reading is a test for button confirmation. The button reading will change from zero to one when the button is pressed and return to 0 when the button is released. Whenever it detects a one, it means a confirmation that can be used in a menu control.

Reinforcement fixing is chosen as the planned MR training to utilize gloves, as many hand movements will be included. In addition, the fixing will be performed at a height inspired by the scaffolding game from Motive Force. Furthermore, a real hoop with a sensor will be used in this MR training. The trainee should perform the reinforcement fixing while standing in the right position; otherwise, a fall of dead animation will be displayed and the trainee will fail the MR training shown in Figure 8 and Figure 9.

The performance result is shown in

Table 12. Performance results.

	Traditional	Mixed Reality Platform
Accuracy
Acceleration and coordinate	92.50%	95.00%
Finger bending	94.00%	95.50%
Confirm button	97.00%	97.75%
Errors ratio
Acceleration and coordinate	13.25%	11.25%
Finger bending	5.00%	4.25%
Confirm Button	0.25%	0.20%
Processing time
Acceleration and Coordinate	8.00 ± 3.0 s	13.00 ± 5.0 s
Finger bending	5.00 ± 2.0 s	7.00 ± 2.5 s
Confirm Button	2.00 ± 1.0 s	2.50 ± 1.5 s

. There are 30 participants involved in the experiments with two sets of testing: the traditional method and an MR-based platform with the glove. Three movements are adopted: acceleration and coordinate, finger bending, and confirm button. Each movement has been randomly generated and repeated ten times. The results show that the MR-based platform resulted in a higher accuracy rate and a lower error rate. Still, the processing time is much longer than the traditional method’s simulation. This might be because the participants are not familiar with the glove or the MR-based platform. VR and AR technology has become more viable over the decade and has started to be applied in different aspects of work such as video games, industrial training, education, etc. On the other hand, the number of occupational injuries in the past decade [1] has shown that OSH has become more important as society advances. Different types of work require training before hiring to ensure employees fulfill their job duty correctly and safely. However, some movements may be challenging due to cost or safety issues. For example, surgical training is costly as different organs and tools are needed for medical students. Site workers who work at height will also require training, but it will be hazardous if the activity occurs at a high location. Therefore, MR technology has become a trend in training aspects as it can simulate different environments, which can be costly or dangerous. In addition, as MR is a program, it can be easily standardized, which is positive when designing a training course. Although MR training is viable and trending nowadays, it still has limitations. Most of the MR controllers are standardized into point-and-click control. It is hard for this control method to simulate precise movement that are used in reality, such as surgical movements. Furthermore, those controllers are mainly made by motion sensors, which cannot affect real situations such as heat or pressure. Therefore, we propose this project to modify the current MR controller and make it more compliant with the actual working environment.

OSH involves workplace safety, health and welfare issues. It aims at providing a better working environment to employees in a modern society by using laws, standards, and programs. A company with good OSH standards can improve its brand image and employee morale. OSH concerns workplace hazards such as chemical, physical, biological, psychological, and accidents. Hence, the workplace is improved the workplace by removing or reducing such risks. Employers must provide a safe and healthy working environment for their employees in modern society by following the principle of OSH. For example, wearing a safety helmet on construction sites is one of the OSH requirements in many construction companies. Complying with good OSH standards in the workplace can ensure employees’ safety and health, boosting their working efficiency. It can also reduce the chance of accidents, thus preventing loss of time in construction sites or even loss of human resources. In addition, a company with good employee welfare can increase its brand image, thus attracting more talented people to join the company and increasing company competitiveness. On the other hand, maintaining a reasonable OSH manner requires personnel to handle related issues. Usually, big companies have a safety, health, and quality department responsible for OSH progress status; extra funds are required. Therefore, OSH can be a burden for small companies and may slow down the work process as everything has OSH standards or rules to follow, and employees may make an extra effort to comply with OSH requirements.

5. Concluding Remarks

Every injury due to an occupational accident is heart-breaking news. It can be commonly agreed that all efforts will improve OSH. We believe that accidents can be avoided with good quality training. Therefore, developing a better-quality MR-based OSH training program is our primary aim, achieved by improving the current MR wearable devices. The results show that the overall accuracy is higher in the traditional training platform rather than the MR-based platform. Still, the participants may need more time to become familiar with the glove. Once they get used to it, the operation time and the error ratio should be lower compared to the traditional method. It is hoped that this can increase the immersive level of the training. Thus, the trainees will better understand potential dangers in their future work. The MR-based platform could further provide suggestions to the trainee for corrections and improvement. Hopefully, it can reduce occupational accidents, especially in the construction industry. In future, more complicated tasks could be designed under the proposed platform and might also include tasks related to human–robot interaction [85,86,87].

The following factors can be improved in the future to enhance the prototype. The first factor that can be improved in the future is the accuracy of the prototype. According to the experiment’s data, the flex sensor’s current bending rate is not accurate enough, and the sensing range of the heartbeat sensor cannot provide the expected data. In order to tackle the issues, using more professional sensors may be a feasible solution. In addition, for the sensing range of the flex sensor, a correction factor should be introduced to reduce the difference among different individual hands. Such a correction factor should be calculated by further study in the future. The second factor that can be improved in the future is the usability of the prototype. In this project, only the prototype of the wearable glove is developed, which has not yet been adopted for the MR-based OSH platform. To improve the usability of our prototype, an MR training program based on the received result (e.g., a training program on reinforcement fixing at height) should be designed and developed in the future. One of the aims is to create a unity 3D hand model linked with the glove and the MR training game. Moreover, to further enhance the usability of the prototype, feedback systems (e.g., vibration, temperature change, etc.) and IoT between different wearable devices (e.g., jackets, shoes, etc.) should be considered in the future. The final factor that can be improved in the future is user experience. The prototype is now constrained by USB wire (work as power supply and data transfer) due to the limitation of time and skill. As an improvement, wireless signal transfer (e.g., Bluetooth and Wi-Fi) can be considered in the future, while a portable power supply (e.g., battery box) can be added to the prototype. Cabling can be improved in the future to minimize hindering users’ movement. Based on the HCPS proposed in this paper, the future work of the cybersecurity could be considered and developed. The data transmission functionality could be developed and adopted by the blockchain for further analysis. Under the environment and framework we proposed and developed, similar gloves could be tested with multiple tasks for ensuring OSH before the actual operation.