Sensor-Based Motion Tracking System Evaluation for RULA in Assembly Task

Industries need a mechanism to monitor the workers’ safety and to prevent Work-related Musculoskeletal Disorders (WMSDs). The development of ergonomics assessment tools helps the industry evaluate workplace design and worker posture. Many studies proposed the automated ergonomics assessment method to replace the manual; however, it only focused on calculating body angle and assessing the wrist section manually. This study aims to (a) propose a wrist kinematics measurement based on unobtrusive sensors, (b) detect potential WMSDs related to wrist posture, and (c) compare the wrist posture of subjects while performing assembly tasks to achieve a comprehensive and personalized ergonomic assessment. The wrist posture measurement is combined with the body posture measurement to provide a comprehensive ergonomics assessment based on RULA. Data were collected from subjects who performed the assembly process to evaluate our method. We compared the risk score assessed by the ergonomist and the risk score generated by our method. All body segments achieved more than an 80% similarity score, enhancing the scores for wrist position and wrist twist by 6.8% and 0.3%, respectively. A hypothesis analysis was conducted to evaluate the difference across the subjects. The results indicate that every subject performs tasks differently and has different potential risks regarding wrist posture.


Introduction
Work-related Musculoskeletal Disorders (WMSDs) are the major outcome of occupational injuries in the industry [1]. The Korean Ministry of Employment and Labor reported 9440 cases of WMSDs in 2019; these case rates increased by 40.6% from the previous year [2]. WMSDs can decrease workers' productivity and increase workers' compensation costs as WMSDs are the main cause of absenteeism, lost working days, and temporary or permanent disability [3,4]. The dominant factors that lead to WMSDs in manual assembly tasks are awkward working postures, repetitive movements, and workloads [5][6][7]. Repetitive movements are a significant risk factor for upper limb WMSDs, especially for the arm and wrist. Some injuries and disorders affect the wrist, such as carpal tunnel syndrome, wrist tendonitis, and lateral epicondylitis [8].
Many studies have proposed a method to help industries evaluate their work environment. There is a trend in measuring technical ergonomics based on sensors, such as wearable inertial sensors [9,10]. Many ergonomic assessment research works rely on a vision-based approach utilizing either image [11,12] or motion sensors like Kinect [13] to focus on high-level activities (e.g., lifting, sitting, kneeling, hammering, and drilling) without considering the Therbligs activities. The most active body part in the assembly process is the hand (i.e., arm, wrist, and fingers). Therbligs motions (e.g., grasp, assemble, inspect, etc.) can be used to describe the low-level activity done by hand, and these motions are related to wrist kinematics [14]. For example, when a worker grasps something, the

•
We propose a geometric-based wrist kinematics measurement for radial-ulnar deviation and wrist twist measurement based on the flexion-extension angle and forearm pronation-supination from the hand tracking data sensor.

•
We present a comprehensive ergonomics assessment (i.e., RULA) using a derived wrist posture measure, along with a body posture measure, in the assembly process automatically.

•
We present an extensive experiment to show a personalized ergonomic assessment using multimodal unobtrusive sensors (i.e., body tracking and hand tracking sensors).

Wrist Kinematics
Some previous studies focused on the wrist assessment for a specific activity, such as laparoscopic surgery [36,37] and activity during mouse use [38]. The authors in [36] measured the flexion-extension angles of the surgeon's wrist using wireless electro-goniometer sensors and CAPTIV version L7000 software. The sensors were placed on the back of the right and left hands and forearms [36]. Ref. [37] utilized five sensors for monitoring the forces exerted by the distal phalanges of the surgeon's thumb; index, middle, and ring fingers; and palm. All data were examined with custom-made software [37]. Ref. [38] used an infrared three-dimensional (3D) motion analysis system (Optotrak Certus, Northern Digital, Ontario, Canada) to measure the posture of the upper extremities; moreover, six-degrees-of-freedom force-torque sensors (ATI, Apex, NC, USA) were attached to each arm support plate and the mouse pad. This study evaluated six support conditions (i.e., forearm support, flat palm support, raised palm support, forearm + flat palm support, forearm + raised palm support, and no support) and the joint angles were calculated from the Euler angles; these are defined by the rotation matrices that describe the orientation of the distal segment relative to the proximal segment concerning the anatomical position, and the vertical position was used to determine the joint angles [38].
Some previous studies estimated wrist kinematics by collecting hand motion data using a biosensor. The author in [18] proposed a novel regression scheme for supervised domain adaptation (SDA), wherein the domain shift effects were effectively reduced to enhance the inter-subject performances of CNN in the wrist kinematics estimation. This study recorded sEMG signals using the Delsys TrignoTM system. The electrodes were placed over five primary wrist muscles in the right forearm: Flexor Carpi Radialis (FCR), Flexor Carpi Ulnaris (FCU), Extensor Carpi Radialis Longus (ECRL), Extensor Carpi Radialis Brevis (ECRB), and Extensor Carpi Ulnaris (ECU) [18]. Another previous work proposed a Convolutional Neural Network (CNN)-Long Short-Term Memory (LSTM) hybrid model to fully explore the temporal-spatial information from surface electromyography (sEMG). The wrist movements were simultaneously recorded with Shimmer wearable sensors, which were attached to the back of the testing hand [17]. Figure 1 shows our ergonomics assessment workflow, which consists of three sections: data preparation, automated RULA, and evaluation. Data preparation has three phases: data collection, data fusion, and data processing. In the automated RULA section, the process starts with the expert labeling the data with a RULA score. The expert assesses the worker's posture by visualizing the data using 3D visualization tools and matches it with the video recorded during data collection. We use the expert-labeled data as ground truth. The next phase is the proposed ergonomics assessment method that evaluates the body posture and wrist kinematics (i.e., flexion-extension, radial-ulnar deviation, and pronation-supination) to determine the RULA score. The last section is the evaluation, wherein we measure the similarity of the RULA scores that were calculated by the expert and automatically calculated by the system. In addition, to evaluate the heterogeneity between subjects, we perform hypothesis testing to justify the personalized measurement.

Data Preparation
The data was collected from 12 subjects-eleven subjects were right-handed and one subject was left-handed. All subjects included in the data collection signed the "Research Subject Confidentiality Agreement" form. We collected the data using a body tracking sensor and a hand tracking sensor. The first sensor is the Orbbec camera, which has a sampling rate of 30Hz and is used to record body movements. The second sensor, the Leap Motion, aims to record hand motions with a 64 Hz sampling rate. Since both sensors have different sampling rates, we have to balance the sampling rates to fuse the data from both sensors. We used Piecewise Aggregate Approximation (PAA) to reduce the dimensionality of the time series to 20 Hz.
Using Ref. [39], we adjusted the data with a 20 Hz sampling rate for the next step. During data collection, we also label the data with three labels that represent the activity levels: high-level, middle-level, and low-level activities.The data are labeled after the dimension reduction process is complete. Furthermore, because RULA requires calculating the joint angles for six body sections, we cannot skip the calculation for any section; therefore, we remove all null value rows.

Labeling by Expert
We validate our automated RULA score calculation using an expert's evaluation. The expert herein means a specialist who understands the whole assembly process, follows the data collection process and has knowledge of ergonomics assessment and risk posture. The expert also has a background in industrial and computer science. The expert-labeled data includes the RULA score and risk level. Since labeling postures for 12 subjects will take a long time, the expert only evaluates and labels the posture for every ten rows of data (per 0.5 s). The posture evaluation is based on 3D visualizations and recorded videos. The 3D visualization is provided through two types of projection: a front-right view (XZYprojection) and back-left view (ZXY-projection). The total number of expert-labeled postures labeled is 4729. The expert assesses the left upper arm, right upper arm, left lower arm, right lower arm, right wrist, left wrist, neck, trunk, and legs for every pose. We

Data Preparation
The data was collected from 12 subjects-eleven subjects were right-handed and one subject was left-handed. All subjects included in the data collection signed the "Research Subject Confidentiality Agreement" form. We collected the data using a body tracking sensor and a hand tracking sensor. The first sensor is the Orbbec camera, which has a sampling rate of 30Hz and is used to record body movements. The second sensor, the Leap Motion, aims to record hand motions with a 64 Hz sampling rate. Since both sensors have different sampling rates, we have to balance the sampling rates to fuse the data from both sensors. We used Piecewise Aggregate Approximation (PAA) to reduce the dimensionality of the time series to 20 Hz.
PAA uses three steps to reduce dimensionality. First, it divides the original time series into N equal-length segments. Second, it computes the mean of each segment. Third, it replaces the value of the segment with its mean. Given this, XT = [XT 1 , XT 2 , . . . , XT k , . . . , XT m ] is a multivariate time series where m denotes the number of variables (1 ≤ k ≤ m); XT k is an n-length univariate time series, where XT k = {xt 1 , xt 2 , . . . , xt i , . . . , xt n }. xt i (1 ≤ i ≤ n) i denotes the value in time; and n is the entire length of the time series XT k . For example, XT 1 = {xt 1 , xt 2 , . . . , xt i , . . . , xt n } is a univariate time series with n-length, and PAA approximates time series XT 1 into XT 1 = xt 1 , xt 2 , . . . , xt i , . . . , xt N , where N is the number of segments [39]. Equation (1) shows the formula xt i for calculating each xt i .
Using Ref. [39], we adjusted the data with a 20 Hz sampling rate for the next step. During data collection, we also label the data with three labels that represent the activity levels: high-level, middle-level, and low-level activities.The data are labeled after the dimension reduction process is complete. Furthermore, because RULA requires calculating the joint angles for six body sections, we cannot skip the calculation for any section; therefore, we remove all null value rows.

Labeling by Expert
We validate our automated RULA score calculation using an expert's evaluation. The expert herein means a specialist who understands the whole assembly process, follows the data collection process and has knowledge of ergonomics assessment and risk posture. The expert also has a background in industrial and computer science. The expert-labeled data includes the RULA score and risk level. Since labeling postures for 12 subjects will take a long time, the expert only evaluates and labels the posture for every ten rows of data (per 0.5 s). The posture evaluation is based on 3D visualizations and recorded videos. The 3D visualization is provided through two types of projection: a front-right view (XZY-projection) and back-left view (ZXY-projection). The total number of expert-labeled postures labeled is 4729. The expert assesses the left upper arm, right upper arm, left lower arm, right lower arm, right wrist, left wrist, neck, trunk, and legs for every pose. We develop a visualization tool to make labeling easier. This tool displays a 3D visualization of the skeleton with the XZY-projection or ZXY-projection. Figure 2 describes the annotation process that the expert uses. develop a visualization tool to make labeling easier. This tool displays a 3D visualization of the skeleton with the XZY-projection or ZXY-projection. Figure 2 describes the annotation process that the expert uses.

RULA Score Calculation
The RULA score is determined based on the body joint angle at each region. Six body regions need to be evaluated: the upper arm, lower arm, wrist, neck, trunk, and legs. There are several aspects that must be considered in the evaluation of each part of the body. For example, the right upper arm evaluation involves the right upper arm position, right shoulder raise, and right upper arm abduction [40]. The first step in determining the upper arm position score is the calculation of the joint angles between the right elbow, right shoulder, and right hip. Afterward, we can determine the upper arm position score according to the RULA worksheet threshold.
RULA divides the body assessment into two sections: section A and section B. Section A consists of the arm and wrist, while section B consists of the neck, trunk, and legs [40]. Both sections generate an intermediate RULA score of A and B, respectively. Afterward, the RULA grand score is calculated by combining both intermediate RULA scores [16]. In this study, intermediate RULA scores A combines the multiple sensor angle calculation. The joint angle of the upper arm and lower arm position is calculated using the data from the body tracking sensor, while the wrist position is calculated using the data from the hand tracking sensor.
We calculated the joint angle from the left and right sides. Furthermore, we calculated a Max score that is defined by the highest score between the left and right position. To determine intermediate RULA score B, we need the score from the neck position, trunk position, and legs. The joint angle of the neck and trunk position are calculated using the data from the body tracking sensor, while the legs score is directly defined (i.e., 2) for all postures given that all of the subjects perform all of the tasks in a standing position, and the legs and feet are not supported. After scores A and B are calculated, we can calculate the grand score (score C). There are three kinds of grand score: the left grand score, which is calculated based on score A from the left side and score B; the right grand score, which is calculated based on score A from the right side and score B; and the general grand score, which is calculated based on the Max score and score B. Figure 3 illustrates the RULA scoring mechanism. The RULA grand score is categorized into four risk levels: negligible, low, medium, and high. Table 2 presents the risk levels with the actions that should be taken for each risk level.  The RULA score is determined based on the body joint angle at each region. Six body regions need to be evaluated: the upper arm, lower arm, wrist, neck, trunk, and legs. There are several aspects that must be considered in the evaluation of each part of the body. For example, the right upper arm evaluation involves the right upper arm position, right shoulder raise, and right upper arm abduction [40]. The first step in determining the upper arm position score is the calculation of the joint angles between the right elbow, right shoulder, and right hip. Afterward, we can determine the upper arm position score according to the RULA worksheet threshold.
RULA divides the body assessment into two sections: section A and section B. Section A consists of the arm and wrist, while section B consists of the neck, trunk, and legs [40]. Both sections generate an intermediate RULA score of A and B, respectively. Afterward, the RULA grand score is calculated by combining both intermediate RULA scores [16]. In this study, intermediate RULA scores A combines the multiple sensor angle calculation. The joint angle of the upper arm and lower arm position is calculated using the data from the body tracking sensor, while the wrist position is calculated using the data from the hand tracking sensor.
We calculated the joint angle from the left and right sides. Furthermore, we calculated a Max score that is defined by the highest score between the left and right position. To determine intermediate RULA score B, we need the score from the neck position, trunk position, and legs. The joint angle of the neck and trunk position are calculated using the data from the body tracking sensor, while the legs score is directly defined (i.e., 2) for all postures given that all of the subjects perform all of the tasks in a standing position, and the legs and feet are not supported. After scores A and B are calculated, we can calculate the grand score (score C). There are three kinds of grand score: the left grand score, which is calculated based on score A from the left side and score B; the right grand score, which is calculated based on score A from the right side and score B; and the general grand score, which is calculated based on the Max score and score B. Figure 3 illustrates the RULA scoring mechanism. The RULA grand score is categorized into four risk levels: negligible, low, medium, and high.

Score
Risk Level Action to Be Taken  1-2 Negligible Acceptable posture if it is not repeated for a longer period 3-4 Low Further investigation and change may be needed in future 5-6 Medium The investigation and change are required soon 7 High The investigation and change are required immediately

Wrist Kinematics Measurement
This section discusses wrist kinematics measurement. The wrist bone structure consists of eight carpal bones that connect proximally to the forearm's radius and connect distally to the hand's five metacarpals. The flexion-extension and radial-ulnar deviation of the wrist are its global motions. The wrist is able to perform those movements due to articulation at the radiocarpal joint, midcarpal joint, the carpo-metacarpal joint, and between individual carpal bones. The forearm pronation-supination motion is possible due to the distal radioulnar joint [21].
RULA involves the evaluation of the wrist flexion-extension, wrist radial-ulnar deviation, and forearm pronation-supination to assess the wrist section. This study assesses the wrist section with the data collected by the finger tracking sensor. Figure 4 shows the finger joints from the finger tracking sensor. Each hand has 25 joints, and all of the joints are represented by a 3D coordinate ( , , ). The finger joints are used to calculate the wrist position and wrist twist score. As shown in Figure 4, we label every finger joint-'L' for the left hand and 'R' for the right hand. A list of the finger joints is provided in Table A1.

Wrist Kinematics Measurement
This section discusses wrist kinematics measurement. The wrist bone structure consists of eight carpal bones that connect proximally to the forearm's radius and connect distally to the hand's five metacarpals. The flexion-extension and radial-ulnar deviation of the wrist are its global motions. The wrist is able to perform those movements due to articulation at the radiocarpal joint, midcarpal joint, the carpo-metacarpal joint, and between individual carpal bones. The forearm pronation-supination motion is possible due to the distal radioulnar joint [21].
RULA involves the evaluation of the wrist flexion-extension, wrist radial-ulnar deviation, and forearm pronation-supination to assess the wrist section. This study assesses the wrist section with the data collected by the finger tracking sensor. Figure 4 shows the finger joints from the finger tracking sensor. Each hand has 25 joints, and all of the joints are represented by a 3D coordinate (x, y, z). The finger joints are used to calculate the wrist position and wrist twist score. As shown in Figure 4, we label every finger joint-'L' for the left hand and 'R' for the right hand. A list of the finger joints is provided in Table A1. Table 3 shows the finger joints that are involved in wrist kinematics measurement. Four wrist kinematics need to be measured: wrist flexion-extension, wrist radial-ulnar deviation, forearm pronation-supination, and the range of the forearm when pronating and supinating.

1.
Wrist flexion-extension The calculation of the wrist's flexion-extension angle involves two 3D points from the middle finger. Figure 5 illustrates the point position from the right hand, where R13 and R14 represent the middle finger proximal phalanges endpoint and middle finger metacarpals endpoint, respectively.
Wrist is at or near the end of the range range of pronation-supination

Score
Risk Level Action to Be Taken  1-2 Negligible Acceptable posture if it is not repeated for a longer period 3-4 Low Further investigation and change may be needed in future 5-6 Medium The investigation and change are required soon 7 High The investigation and change are required immediately

Wrist Kinematics Measurement
This section discusses wrist kinematics measurement. The wrist bone structure consists of eight carpal bones that connect proximally to the forearm's radius and connect distally to the hand's five metacarpals. The flexion-extension and radial-ulnar deviation of the wrist are its global motions. The wrist is able to perform those movements due to articulation at the radiocarpal joint, midcarpal joint, the carpo-metacarpal joint, and between individual carpal bones. The forearm pronation-supination motion is possible due to the distal radioulnar joint [21].
RULA involves the evaluation of the wrist flexion-extension, wrist radial-ulnar deviation, and forearm pronation-supination to assess the wrist section. This study assesses the wrist section with the data collected by the finger tracking sensor. Figure 4 shows the finger joints from the finger tracking sensor. Each hand has 25 joints, and all of the joints are represented by a 3D coordinate ( , , ). The finger joints are used to calculate the wrist position and wrist twist score. As shown in Figure 4, we label every finger joint-'L' for the left hand and 'R' for the right hand. A list of the finger joints is provided in Table A1.   Table 3 shows the finger joints that are involved in wrist kinematics measurem wrist kinematics need to be measured: wrist flexion-extension, wrist radial-ulnar d forearm pronation-supination, and the range of the forearm when pronating and ing.
Wrist is at or near the end of the range range of pronationsupination

Wrist flexion-extension
The calculation of the wrist's flexion-extension angle involves two 3D poi the middle finger. Figure 5 illustrates the point position from the right hand, w and R14 represent the middle finger proximal phalanges endpoint and middle fin acarpals endpoint, respectively. In this case, ( 1 , 1 , 1 ) is the first point and ( 2 , 2 , 2 ) is the second poin tion (2) shows the formula that is used to calculate the angle of wrist position [41 We also assess the wrist radial-ulnar deviation (wrist bending) to adjust th In this case, a(x 1 , y 1 , z 1 ) is the first point and b(x 2 , y 2 , z 2 ) is the second point. Equation (2) shows the formula that is used to calculate the angle of wrist position [41].

Wrist radial-ulnar deviation
We also assess the wrist radial-ulnar deviation (wrist bending) to adjust the RULA score for the wrist. We assume that if the wrist is bent 15 • or more, the wrist score will increase by 1. The joint involved in assessing wrist bending is (R05, R04, R24) for the right hand and (L05, L04, L24) for the left hand. The three joint points represent the position of the start point of the index finger of the distal phalanges, the metacarpals endpoint of the thumb, and the metacarpals endpoint of the pinky finger. The illustration of the assessment of the wrist's radial-ulnar deviation angle is shown in Figure 6. The deviation angle is the angle between two vectors that is formed by three joint points, where each joint consists of 3D coordinates ( , , ). In this case, ( 1 , 1 , 1 ), ( 2 , 2 , 2 ), and ( 3 , 3 , 3 ) represent the first, second, and third joint positions, respectively. Equation (3) shows the formula that is used to calculate the joint angle.

Forearm pronation-supination
The next position that should be assessed is the pronation-supination (wrist twist). Pronation is the rotation of the hand from the handshake position to the position where the palm is facing downward. In contrast, supination is the rotation of the hand from the handshake position to an upward-facing palm surface position [40]. Figure 7 illustrates wrist twists in mid-range (a) and the wrist at or near the end of range (b). To evaluate the wrist twist, we calculate the angle between two normal vectors. ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , and ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ represent the normal vectors of the initial position of the right hand, the current position of the right hand, the initial position of the left hand, and the current position of the left hand, respectively. The normal vector is a function that is used to find the normal vector from the coronal plane. Suppose ( 1 , 1 , 1 ), ( 2 , 2 , 2 ), and ( 3 , 3 , 3 ) represent three joint positions, then the equation that is used to find the normal vector is defined as shown in Equation (4) [41].
Joints L13, L04, and L24 are involved in finding the normal vector for the left hand, while the normal vector for the right hand is defined by joints R13, R04, and R24. Figure The deviation angle is the angle between two vectors that is formed by three joint points, where each joint consists of 3D coordinates (x, y, z). In this case, a(x 1 , y 1 , z 1 ), b(x 2 , y 2 , z 2 ), and c(x 3 , y 3 , z 3 ) represent the first, second, and third joint positions, respectively. Equation (3) shows the formula that is used to calculate the joint angle.

Forearm pronation-supination
The next position that should be assessed is the pronation-supination (wrist twist). Pronation is the rotation of the hand from the handshake position to the position where the palm is facing downward. In contrast, supination is the rotation of the hand from the handshake position to an upward-facing palm surface position [40]. Figure 7 illustrates wrist twists in mid-range (a) and the wrist at or near the end of range (b). To evaluate the wrist twist, we calculate the angle between two normal vectors. The normal vector is a function that is used to find the normal vector from the coronal plane. Suppose a(x 1 , y 1 , z 1 ), b(x 2 , y 2 , z 2 ), and c(x 3 , y 3 , z 3 ) represent three joint positions, then the equation that is used to find the normal vector is defined as shown in Equation (4) [41].
Joints L13, L04, and L24 are involved in finding the normal vector for the left hand, while the normal vector for the right hand is defined by joints R13, R04, and R24. Figure 8 is an illustration of the wrist twist (pronation-supination) angle calculation. The angle between two planes is calculated to evaluate the pronation-supination position. The angle is determined between two normal vectors from two planes. In this case, the two normal vectors are determined by the initial position ( − −− → IPR nv and − −− → IPL nv ) and the current position Pronation is the rotation of the hand from the handshake position to the position where the palm is facing downward. In contrast, supination is the rotation of the hand from the handshake position to an upward-facing palm surface position [40]. Figure 7 illustrates wrist twists in mid-range (a) and the wrist at or near the end of range (b). To evaluate the wrist twist, we calculate the angle between two normal vectors. ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ , and ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ represent the normal vectors of the initial position of the right hand, the current position of the right hand, the initial position of the left hand, and the current position of the left hand, respectively.
(a) (b) Figure 7. Wrist twist assessment: (a) wrist is twisted in mid-range (pronation-supination); (b) wrist is at or near the end of range.
The normal vector is a function that is used to find the normal vector from the coronal plane. Suppose ( 1 , 1 , 1 ), ( 2 , 2 , 2 ), and ( 3 , 3 , 3 ) represent three joint positions, then the equation that is used to find the normal vector is defined as shown in Equation (4) [41].
Joints L13, L04, and L24 are involved in finding the normal vector for the left hand, while the normal vector for the right hand is defined by joints R13, R04, and R24. Figure  8 is an illustration of the wrist twist (pronation-supination) angle calculation. The angle between two planes is calculated to evaluate the pronation-supination position. The angle is determined between two normal vectors from two planes. In this case, the two normal vectors are determined by the initial position ( ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ and ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ ) and the current position ( ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ and ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ ). For adjusting the wrist twist posture assessment, Figure 9 illustrates the evaluation of the wrist twist posture in the flexion-extension range. The angle calculation for this assessment is also used for the calculation between the two normal vectors that are produced from the initial position plane and the current position plane. Equations (5) and (6) define the formula that is used to calculate the angle of the wrist twist for the right and left hand, respectively.

Body Posture Measurement
The measurement method for body posture (i.e., upper arm, lower arm, neck, trunk, and legs) is discussed in this section. Figure 10 shows the body joints from the body tracking sensor. There are 16 body joints, and each joint is represented by a 3D coordinate ( , , ). A list of the body joints is provided in Table A2. The body joints are used to calculate the score for the upper arm, lower arm, neck, and trunk. For adjusting the wrist twist posture assessment, Figure 9 illustrates the evaluation of the wrist twist posture in the flexion-extension range. The angle calculation for this assessment is also used for the calculation between the two normal vectors that are produced from the initial position plane and the current position plane. Equations (5) and (6)  For adjusting the wrist twist posture assessment, Figure 9 illustrates the evaluation of the wrist twist posture in the flexion-extension range. The angle calculation for this assessment is also used for the calculation between the two normal vectors that are produced from the initial position plane and the current position plane. Equations (5) and (6) define the formula that is used to calculate the angle of the wrist twist for the right and left hand, respectively.

Body Posture Measurement
The measurement method for body posture (i.e., upper arm, lower arm, neck, trunk, and legs) is discussed in this section. Figure 10 shows the body joints from the body tracking sensor. There are 16 body joints, and each joint is represented by a 3D coordinate ( , , ). A list of the body joints is provided in Table A2. The body joints are used to calculate the score for the upper arm, lower arm, neck, and trunk.

Body Posture Measurement
The measurement method for body posture (i.e., upper arm, lower arm, neck, trunk, and legs) is discussed in this section. Figure 10 shows the body joints from the body tracking sensor. There are 16 body joints, and each joint is represented by a 3D coordinate (x, y, z). A list of the body joints is provided in Table A2. The body joints are used to calculate the score for the upper arm, lower arm, neck, and trunk.  Table 4 shows the body joints that are involved in angle calculations. Since ( 1 , 1 , 1 ), ( 2 , 2 , 2 ), and ( 3 , 3 , 3 ) represent the first, second, and third joint positions, respectively; the calculation of the joint angle is performed by Equation (2). Since the subjects are working in standing positions without any support during the assembly process, the legs score is determined to be 2. Furthermore, we do not consider the neck twist, muscle, or force score.   Table 4 shows the body joints that are involved in angle calculations. Since a(x 1 , y 1 , z 1 ), b(x 2 , y 2 , z 2 ), and c(x 3 , y 3 , z 3 ) represent the first, second, and third joint positions, respectively; the calculation of the joint angle is performed by Equation (2). Since the subjects are working in standing positions without any support during the assembly process, the legs score is determined to be 2. Furthermore, we do not consider the neck twist, muscle, or force score.
The NV() is a function that is used to find the normal vector from the coronal plane. The equation that is used to find the normal vector is defined as shown in Equation (4) [41].

Evaluation
To evaluate the proposed method, this study compares the score from expert observation with that calculated by our method; moreover, we measure the similarity of the RULA score on each high-level activity. Since ES = es 1 , es 2 , . . . , es n is the list of scores from expert observation and SS = ss 1 , ss 2 , . . . , ss n is the list of the scores from system calculation, Equation (7) shows the similarity numSim(es, ss) between two numbers, es and ss. Then, Equation (8) is the formula that is used to calculate the mean similarity tsim(ES, SS). The intervals for the similarity measurement are [0, 1], where 1 indicates the maximum similarity [42].
numSim(es, ss) = 1 − |es − ss| |es| + |ss| (7) tsim(ES, SS) = 1 n n ∑ a=1 numSim(es i , ss i ) Furthermore, a hypothesis analysis was also conducted to evaluate and compare the differences in wrist posture across the subjects. This analysis was performed using a one-way ANOVA and t-test.

Laboratory Setup
The data were collected from 12 subjects. Every subject had to follow the assembly steps that were given by the instructor. Figure 11 shows the laboratory setup during data collection. We used two sensors to record the assembly activity. The Orbecc Astra Pro camera collected body movement data from the subjects during assembly; it was placed in front of the subjects. The second sensor was Leap Motion, which was used to collect finger movement during the assembly process. It was located on the table, and we ensured that the sensor could cover the entire assembly area. There was also a monitor that showed the skeleton movements from the Orbecc and Leap Motion sensors. This laboratory setup was also used for training sessions by the subjects to train in the assembly process before the data collection process. Figure 12 shows the product we assembled (a) and the panel positioning we used during the data collection (b). The size of the product is 30 × 28 × 31, which consists of main panels, side panels, wooden pins, and a drawer. The height of the table is 72 cm. We collected the data from 12 subjects with a height range of 155-178 cm. The average subject's height was 170 cm. Each subject assembled the product three times, and we chose the best performance based on the number of null values during the assembly process.
The process for drawer assembly is divided into five main steps: assemble side panel, assemble main panel, integrate panel, prepare the workspace, and slide the mid-panel. Figure 13 shows the list of assembly activities with three levels of activity. The low level is represented by the Therblig activity. Therblig is a system for assessing the motions required to complete a task, and it is able to examine the smallest of motions. There are 18 motions that can be used to describe the task [43,44]. This study only uses the six Therbligs that are the most relevant to our assembly task (i.e., grasp, transport loaded, assemble, inspect, position, and release load). The middle-level label aims to bridge the low-level activities and high-level activities.

Laboratory Setup
The data were collected from 12 subjects. Every subject had to follow the assembly steps that were given by the instructor. Figure 11 shows the laboratory setup during data collection. We used two sensors to record the assembly activity. The Orbecc Astra Pro camera collected body movement data from the subjects during assembly; it was placed in front of the subjects. The second sensor was Leap Motion, which was used to collect finger movement during the assembly process. It was located on the table, and we ensured that the sensor could cover the entire assembly area. There was also a monitor that showed the skeleton movements from the Orbecc and Leap Motion sensors. This laboratory setup was also used for training sessions by the subjects to train in the assembly process before the data collection process. Figure 11. Laboratory Setup. Figure 12 shows the product we assembled (a) and the panel positioning we used during the data collection (b). The size of the product is 30 × 28 × 31, which consists of main panels, side panels, wooden pins, and a drawer. The height of the table is 72 cm. We collected the data from 12 subjects with a height range of 155-178 cm. The average The process for drawer assembly is divided into five main steps: assemble side panel, assemble main panel, integrate panel, prepare the workspace, and slide the mid-panel. Figure 13 shows the list of assembly activities with three levels of activity. The low level is represented by the Therblig activity. Therblig is a system for assessing the motions required to complete a task, and it is able to examine the smallest of motions. There are 18 motions that can be used to describe the task [43,44]. This study only uses the six Therbligs that are the most relevant to our assembly task (i.e., grasp, transport loaded, assemble, inspect, position, and release load). The middle-level label aims to bridge the low-level activities and high-level activities. Figure 13. List of Assembly Activity. Figure 14 shows an example of one of the activities in the assembly process in our The process for drawer assembly is divided into five main steps: assemble side panel, assemble main panel, integrate panel, prepare the workspace, and slide the mid-panel. Figure 13 shows the list of assembly activities with three levels of activity. The low level is represented by the Therblig activity. Therblig is a system for assessing the motions required to complete a task, and it is able to examine the smallest of motions. There are 18 motions that can be used to describe the task [43,44]. This study only uses the six Therbligs that are the most relevant to our assembly task (i.e., grasp, transport loaded, assemble, inspect, position, and release load). The middle-level label aims to bridge the low-level activities and high-level activities.  Figure 14 shows an example of one of the activities in the assembly process in our data. The "assemble the main panel" activity consists of three steps. The first step is to pick the panel. The "pick panel" activity consists of two activities: "Grasp" and "Transport Loaded". The second step is the "preparation" activity, which involves inspecting the panel position; therefore, the panel is ready to be installed on the main panel. The last step is to assemble the main panel.  Figure 14 shows an example of one of the activities in the assembly process in our data. The "assemble the main panel" activity consists of three steps. The first step is to pick the panel. The "pick panel" activity consists of two activities: "Grasp" and "Transport Loaded". The second step is the "preparation" activity, which involves inspecting the panel position; therefore, the panel is ready to be installed on the main panel. The last step is to assemble the main panel.

Evaluation Results
This section conveys two different subsections to show the results of our experim through two lenses: similarity measurement and personalized measurement. The sim ity measurement takes place based on the respective body parts. The personalized m urement considers the statistical hypothesis in order to justify that the proposed mea ment could be utilized as a personalized measure.

Similarity Measurement
The evaluation of proposed method is evaluated by measuring the similaritie tween the scores calculated by the expert and the scores calculated by the system evaluation is divided for each high-level activity (i.e., assemble side panel, assemble panel, integrate panel, prepare the workspace, and slide the mid-panel) and is bas the mechanism in Figure 3. We divided the evaluation results into five tables (Tab 9).   Table 6. Similarity measurement results for section A (Max score).

Evaluation Results
This section conveys two different subsections to show the results of our experiments through two lenses: similarity measurement and personalized measurement. The similarity measurement takes place based on the respective body parts. The personalized measurement considers the statistical hypothesis in order to justify that the proposed measurement could be utilized as a personalized measure.

Similarity Measurement
The evaluation of proposed method is evaluated by measuring the similarities between the scores calculated by the expert and the scores calculated by the system. The evaluation is divided for each high-level activity (i.e., assemble side panel, assemble main panel, integrate panel, prepare the workspace, and slide the mid-panel) and is based on the mechanism in Figure 3. We divided the evaluation results into five tables (Tables 5-9). Table 5. Similarity measurement results for section A (right and left).   In the first table (Table 5), we present the similarities between the section A (i.e., upper arm, lower arm, wrist, and wrist twist) scores for the left and right sides. The similarity score for the left side is higher than that for the right side. A possible reason for this is the right hand is more active than the left hand; therefore, the left hand is more likely to stay in the same position.

High-Level Activity
The second table (Table 6) shows the similarities for the section A scores when we use the highest score between the right or left score (Max score) for the upper arm, lower arm, and wrist. As a comparison with the previous study, we measured the similarities between the wrist position and wrist twist score, which were calculated by the previous study method [19], then compared it with the similarities score from the current study method. The similarities results can be seen in the third table (Table 7). Beforehand, the previous study had not performed a score comparison analysis and only considered the higher score (Max score) from the two sides. Other than that, the previous study did not address the adjustment score (i.e., radial-ulnar deviation and range of pronation-supination) and instead relied on the ergonomist's evaluation. Based on the results, the current study enhances the performance of the ergonomics assessment; it improves the similarity scores for the wrist position and wrist twist by 6.8% and 0.3%, respectively. The slight difference in scores for wrist twist is due to the score on wrist twist being measured as only 1 or 2. Table 1 shows the previous studies related to ergonomics assessment; however, most of them only focused on body assessments and the wrist score was manually assessed by an expert (ergonomists). Moreover, some previous studies did not explain their wrist assessment. Therefore, we can only compare our proposed method with one previous study [19]. We also had difficulty implementing our method with another dataset because no public dataset provided a RULA score.
The fourth table (Table 8) presents the evaluation for section B (i.e., neck, trunk, and legs). Due to all of the subjects performing all of the tasks in the standing position, the legs score is directly defined (i.e., 2) for all postures. Therefore, we only present the similarities results for the neck and trunk. The "integrate panel" activity has the lowest similarity. A possible reason for this is that the "integrate panel" activity is a difficult task to perform, and each subject performs the task differently. Compared to our previous study, the current study has an 82% average similarity score for the neck, while the method in our previous study was only 77.2%; this means that changing the torso joint point to the waist joint point can improve the ergonomic assessment performance.
The last table (Table 9) presents the similarities in the RULA grand score. In the experiment, we calculated three kinds of RULA grand scores. First, the Right RULA grand score is calculated based on the score in section B and the right-side score from section A. Second, the Left RULA grand score is calculated from the section B score and the left-side score of section A. Third, considering that the higher the RULA score the higher the risk level, we calculated the General RULA grand score from section B and the Max score from section A using the mechanism in Figure 3. Based on the results, the general grand scores are higher than the right and left sides.

Personalized Measurement
This section evaluates the significant differences among the RULA of all subjects with a one-way ANOVA and t-test. The null hypothesis is H 0 : µ 1 = µ 2 = µ 3 = . . . = µ s , where s is the number of subjects and α = 0.05. The p-value for the ANOVA result is 1.1078 × 10 −41 , which shows that H 0 is rejected. This result justifies that there is at least one subject that is different from other subjects. For a more in-depth analysis, we performed a t-test on every pair of subjects.
Tables 10 and 11 present the t-test results for subjects 1-12. Most of the results are significant, which means that there are differences between subjects in performing the tasks; this shows that even though the subject follows the instructions and has a training session, every subject performs the tasks in their own way. The underlined numbers are the non-significant differences results, which means that they were similarly performed assembly tasks.

Discussion
In this section, we discuss the results of the evaluation methods based on the high-level activities. Figure 15 shows the distribution of the RULA grand scores from each high-level activity. Each activity has a different number of samples. Table 12 indicates the number of samples from the 12 subjects, wherein the RULA grand scores were generated using this system.

Discussion
In this section, we discuss the results of the evaluation methods based on the highlevel activities. Figure 15 shows the distribution of the RULA grand scores from each highlevel activity. Each activity has a different number of samples. Table 12 indicates the number of samples from the 12 subjects, wherein the RULA grand scores were generated using this system.  Based on the similarity measurement results, the overall similarity scores achieve more than 80% similarity. The high similarity scores for wrist position (i.e., flexion-extension and radial-ulnar deviation) and wrist twist (i.e., pronation-supination and the adjustment posture) indicate that our proposed method can measure wrist kinematics and detect potential risk factors; moreover, it improved the performance of ergonomics assessment similarity scores for wrist position and wrist twist by 6.8% and 0.3% on average.
Regarding the high-level activities, the activity "Assemble side panel" has the highest average score. This activity consists of simple tasks, such as attaching the wooden pin to the side panel. It can be seen from Figure 15 that this activity has the fewest high-risk RULA scores. On the contrary, the "Integrate panel" activity has the lowest average score. Possible reasons for this could be that this activity involves a difficult task. This can be seen from Figure 15, which indicates that the activity has the highest number of high-risk  Based on the similarity measurement results, the overall similarity scores achieve more than 80% similarity. The high similarity scores for wrist position (i.e., flexion-extension and radial-ulnar deviation) and wrist twist (i.e., pronation-supination and the adjustment posture) indicate that our proposed method can measure wrist kinematics and detect potential risk factors; moreover, it improved the performance of ergonomics assessment similarity scores for wrist position and wrist twist by 6.8% and 0.3% on average.
Regarding the high-level activities, the activity "Assemble side panel" has the highest average score. This activity consists of simple tasks, such as attaching the wooden pin to the side panel. It can be seen from Figure 15 that this activity has the fewest high-risk RULA scores. On the contrary, the "Integrate panel" activity has the lowest average score.
Possible reasons for this could be that this activity involves a difficult task. This can be seen from Figure 15, which indicates that the activity has the highest number of high-risk RULA scores; moreover, the Orbbec sensor cannot properly capture the joint data because the view is blocked by the drawer when the subject attaches a side panel to the main panel, especially considering the data for the arm position (see Figure 16). A RULA score is determined from the scores from the six body regions. The score for each body region is determined based on the joint angle. The joint angle is calculated based on the position of the body joints that is represented by a 3D coordinate (x, y, z). The 3D coordinate for each body joint is detected and recorded by the Orbbec sensor. Therefore, if the Orbbec sensor cannot properly capture the joint position data, it will affect the evaluation results. Furthermore, the difference in how workers install panels can also affect the result-i.e., sometimes the sensor view for the left hand is blocked, and sometimes the right hand is blocked. The Orbbec sensor actually works well for tracking body motions. Furthermore, since the Orbbec sensor is an unobtrusive sensor, workers can do their tasks without being disturbed by sensors being attached to their bodies; however, if there is a possibility of obstruction, it might be better to use more than one sensor and place it on the other side. RULA scores; moreover, the Orbbec sensor cannot properly capture the joint data because the view is blocked by the drawer when the subject attaches a side panel to the main panel, especially considering the data for the arm position (see Figure 16). A RULA score is determined from the scores from the six body regions. The score for each body region is determined based on the joint angle. The joint angle is calculated based on the position of the body joints that is represented by a 3D coordinate (x, y, z). The 3D coordinate for each body joint is detected and recorded by the Orbbec sensor. Therefore, if the Orbbec sensor cannot properly capture the joint position data, it will affect the evaluation results. Furthermore, the difference in how workers install panels can also affect the result-i.e., sometimes the sensor view for the left hand is blocked, and sometimes the right hand is blocked. The Orbbec sensor actually works well for tracking body motions. Furthermore, since the Orbbec sensor is an unobtrusive sensor, workers can do their tasks without being disturbed by sensors being attached to their bodies; however, if there is a possibility of obstruction, it might be better to use more than one sensor and place it on the other side. Based on the statistical hypothesis testing, it showed that there is a significant difference among most of the subjects (refer to Tables 10 and 11). Some possible factors that impact this are handedness and work duration. In terms of handedness, some subjects claimed to be left-handed. For example, subject 5 and subject 6 had different dominant hands (subject 5 was left-handed while subject 6 was right-handed) and showed significant differences in the statistical hypothesis-although they have similar heights (170 and 168 cm) and have almost the same work duration (4.33 and 4.28 min). In terms of work duration, subject 9 and 10 showed a significant difference. Although both subjects have similar heights (170 and 166 cm, respectively) and both are right-handed, subject 9 was the subject with the fastest working time (3.63 min), while subject 10 had the longest working time (6.48 min); this can indicate that significant differences occur due to the handedness and work duration.
Some results showed an insignificant difference. For example, subject 5 and subject 11 had insignificant differences in their results, which means there was no difference in their risk. A possible reason for this is that both subjects are left-handed, which can indicate similar work patterns. A similar work duration can possibly cause an insignificant difference. For example, subject 4 and subject 5 have similar work durations-4.32 and 4.33 min, respectively.
The risk assessment method in this study can help industries evaluate workplace design to make workers work more efficiently. Furthermore, the method can also provide evaluation methods and suggestions to workers related to their posture in order to increase health awareness and prevent injuries.
In our experiment, the calculation of the RULA score is separate from data collection. We first collected the data from the 12 subjects, then combined all the data. During data collection, the joint can be detected and recorded in real time. We also cleaned the data before calculating the RULA score. As explained in Section 3.3.1, the calculation of the RULA score starts from the joint angle calculation for every body section and is used to determine the score based on the joint angle. Afterward, the calculation process continues Based on the statistical hypothesis testing, it showed that there is a significant difference among most of the subjects (refer to Tables 10 and 11). Some possible factors that impact this are handedness and work duration. In terms of handedness, some subjects claimed to be left-handed. For example, subject 5 and subject 6 had different dominant hands (subject 5 was left-handed while subject 6 was right-handed) and showed significant differences in the statistical hypothesis-although they have similar heights (170 and 168 cm) and have almost the same work duration (4.33 and 4.28 min). In terms of work duration, subject 9 and 10 showed a significant difference. Although both subjects have similar heights (170 and 166 cm, respectively) and both are right-handed, subject 9 was the subject with the fastest working time (3.63 min), while subject 10 had the longest working time (6.48 min); this can indicate that significant differences occur due to the handedness and work duration.
Some results showed an insignificant difference. For example, subject 5 and subject 11 had insignificant differences in their results, which means there was no difference in their risk. A possible reason for this is that both subjects are left-handed, which can indicate similar work patterns. A similar work duration can possibly cause an insignificant difference. For example, subject 4 and subject 5 have similar work durations-4.32 and 4.33 min, respectively.
The risk assessment method in this study can help industries evaluate workplace design to make workers work more efficiently. Furthermore, the method can also provide evaluation methods and suggestions to workers related to their posture in order to increase health awareness and prevent injuries.
In our experiment, the calculation of the RULA score is separate from data collection. We first collected the data from the 12 subjects, then combined all the data. During data collection, the joint can be detected and recorded in real time. We also cleaned the data before calculating the RULA score. As explained in Section 3.3.1, the calculation of the RULA score starts from the joint angle calculation for every body section and is used to determine the score based on the joint angle. Afterward, the calculation process continues with the calculation of the intermediate score (RULA score of section A and section B). The last step is the calculation of the RULA grand score, then saving all of the calculation results.
The total number of expert-labeled postures is 4729. After we measured the similarity between the RULA scores calculated by the expert and the scores calculated by the system, we generated the score for all poses from the 12 subjects. The total number of poses is 54,321. Our risk assessment method required around 0.043156 s to calculate the RULA grand score. This indicates that our method can be implemented for an ergonomics assessment system to evaluate poses in real time. The details of the execution time can be seen in Table A3.
Our method focuses on the development of ergonomics assessments for monitor the safety of workers and the prevention of WMSDs; however, other related domains that use ergonomics assessments, such as elderly health monitoring, worker efficiency management, and medical training, can also benefit from applying our proposed method.
A limitation of this study is that the evaluation of the method only used laboratory data. The muscle and force scores were not considered because the sensors only collected data on the joint positions. A comparison with the existing method is limited, given that no previous studies adopted a combination of body angle calculations and wrist posture assessments. Furthermore, due to the different devices used to collect the data, there is a dissimilarity in data features from the previous studies.

Conclusions
This study proposed a geometric-based wrist kinematics measurement to detect potential risk factors related to wrist position and wrist twist. Our study is able to provide an ergonomics assessment that combines wrist kinematics measurements with body posture measurements during an assembly process based on multimodal unobtrusive sensors (i.e., body tracking and hand tracking sensors). An extensive experiment and evaluation are provided to evaluate our proposed method. The evaluation results show that the current study improves the performance of the ergonomics assessment for wrist position and wrist twist. In addition, the personalized measurement results show that every subject is unique, and even though the subject has a training session before data collection, each subject still has differences from other subjects; this shows that humans perform the same task in their own way and have different potential risks.
In future research, we plan to evaluate our method by extracting data from a realworld industrial environment, analyze the sensitivity of the RULA score changes in similar postures with a slight difference in joint angle, and compare them with other ergonomics assessment tools. In addition, we also plan to build a warning system that provides an alert to a worker if they're in a high-risk posture. This system can be used for assessing training, especially with new workers.