HRC Metrology: Assessment Criteria, Metrics and Methods for Human–Robot Co-Manipulation Tasks

Abstract

We developed a human–robot collaborative manipulation system (co-manipulation system) in the form of a power assist robotic system (PARS) where a human and a robot collaborated to perform the co-manipulation of an object with power assistance. We conducted an experiment (the first experiment), where in each trial of the experiment, a human subject performed the co-manipulation of the object with the PARS, and an expert human–robot co-manipulation researcher observed the co-manipulation task. We collected the co-manipulation and observation data, analyzed the data, and conducted reviews of the related literature, and developed the HRC (human–robot collaboration) metrology, which consisted of necessary criteria, metrics and methods to assess human–robot collaborative manipulation tasks. The proposed HRC metrology consisted of both human–robot collaborative performance and human–robot interactions (HRI) related assessment criteria. Then, we developed another human–robot co-manipulation system using a robot manipulator. In this system, the human–robot co-manipulation task was performed in conjunction with a collaborative assembly task between the robot and human co-workers. In another experiment (the second experiment), we assessed the co-manipulation task for each robotic system separately based on the developed HRC metrology (set of assessment criteria, metrics and methods) to verify and validate the practicality, usability and effectiveness of the criteria, metrics and methods. The results showed that the HRC metrology was effective and practical in assessing the co-manipulation tasks. We then discussed the strengths and limitations of the assessment criteria, metrics and methods. The proposed HRC metrology can be used to assess human–robot collaborative performance and human–robot interactions in human–robot co-manipulation tasks with potential real-world applications in industrial manipulation and manufacturing, transport, logistics, civil construction, rescue and disaster management, timber processing, etc.

1. Introduction

We observe real-world scenarios where human workers need to perform object manipulation tasks [1]. Workers can perform manipulation of objects manually [2]. Workers may also take advantage of existing automated or semi-automated systems (e.g., robotic manipulators, intelligent cranes, etc.) for object manipulation [3,4,5]. Literature reviews and experiences indicate that appropriate human–robot collaboration (HRC) strategies may be more beneficial for object manipulation over manual or fully automated manipulation in terms of flexibility, manipulation quality and workmanship, etc. [6,7,8]. We can implement such collaborative manipulation tasks in manufacturing industries where humans and robots can collaboratively manipulate heavy objects (parts) during assembly operations, e.g., a car door assembly with the car body in an automotive manufacturing factory, warehousing and transportation of heavy assembly materials, etc. [6,7,8]. It is possible to employ various human–robot collaboration methods and strategies to manipulate objects in similar industrial applications (e.g., construction, wood processing, transport, logistics, etc.) [9,10,11]. In some cases, it is possible to use skill and power assistance-type human–robot collaborative methods and strategies to manipulate heavy objects, which can help obtain higher mechanical advantage and task assistance [12].

Appropriate and effective criteria, metrics and methods for assessing HRC in manipulation tasks (co-manipulation tasks) are important considering their benefits and advantages in many aspects, as follows [13,14,15,16,17]. It seems to be advantageous to use the assessment criteria, metrics and methods for generalizing human–robot co-manipulation system design and development. Such generalization may be helpful to compare performance and interactional effectiveness of human–robot co-manipulation tasks and systems [6,7,8]. Appropriately developed metrics and methods can help measure HRC manipulation performance and human–robot interactions easily, make HRC manipulation performance and interactions transparent, compare the performance and human interactions among HRC manipulation systems, and make HRC manipulation design, architecture, and appearance exchangeable [13,14,15,16,17]. The generalization processes of performance criteria and benchmarking practices may facilitate purchase and placement decisions of HRC manipulation systems, which can ensure sustainable and broader implementation and adoption of HRC systems in object manipulation [14]. Nonetheless, enough efforts toward bringing innovations in this research direction are not reported in the literature, except some initial work [13,14,15,16,17].

As we see in the literature, the existing methods and strategies to assess human–robot co-manipulation tasks are limited in scope and applicability, the metrics are not well-defined, and the assessment methods are also not well described. As a result, these metrics and methods cannot facilitate a comprehensive and wholistic assessment and the benchmarking of HRC manipulation performance and interactional effectiveness [13,14,15,16,17]. Due to having limitations in assessment metrics and methods, it is also not so advantageous to use the existing assessment results to compare and benchmark among different HRC manipulation tasks and systems. The state-of-the-art methods to assess human–robot co-manipulation tasks have not been evaluated properly and sufficiently to prove their effectiveness, validity and generality [13,14,15,16,17]. An initiative was taken in this research direction in the previous work in [17]. However, the presented criteria, metrics and methods focused on a weight perception-based power-assisted manipulation scenario, the criteria were not comprehensive for general human–robot collaborative manipulation scenarios, and the metrics and methods were not verified and validated for their appropriateness and effectiveness. Initiatives were taken to develop assessment schemes for object co-manipulation tasks between robots and humans in the past [18,19]. However, each of those past efforts did not address all assessment criteria, metrics and methods, and thus those efforts could not make the assessment scheme comprehensive [18,19]. Instead, the past contributions addressed the criteria, metrics and methods partly, and the assessment scheme was not adequately verified and validated experimentally [17,18,19]. We realize that the assessment criteria need to be divided into different themes and categories such as core HRC performance and physical and cognitive human–robot interactions [8], and the metrics, methods and procedures of assessment of each criterion needs to be presented clearly and adequately [17,18,19]. The scheme should also be verified and validated experimentally.

To address the above limitations and gaps in the state-of-the-art research results and to expand on the initial contributions made in [17,18,19] in this research direction, the objective of this paper is to derive a set of comprehensive assessment criteria, metrics and methods for human–robot collaborative manipulation tasks and ensure their practicality, usability and effectiveness through appropriate verification and validation studies. To do so, in this article we developed a proof-of-concept HRC manipulation system in the form of a power assist robotic system where a human and a robot collaboratively performed the co-manipulation of an object. We then derived a comprehensive set of performance and interaction assessment criteria, metrics and associated methods (i.e., HRC metrology) for human–robot co-manipulation tasks based on human subjects’ and relevant researchers’ experiences with the manipulation tasks and systems and literature reviews. We then developed another human–robot object manipulation system in conjunction with a human–robot assembly task setup. We then assessed both HRC systems for object co-manipulations based on the HRC metrology we derived to verify and validate the effectiveness and practicality of the HRC metrology. We also discussed the strengths and limitations of the HRC metrology. Such an HRC metrology can make HRC manipulation in assembly in manufacturing and other HRC manipulation tasks more transparent, replicable, and comparable [8].

The organization of the paper is as the following: Section 2 accumulates relevant theories and concepts related to HRC manipulation, Section 3 analyzes related work, Section 4 presents the materials and methods for the research including the experimental setups, experimental procedures and experimental results, Section 5 contains a general discussion on the results, and Section 6 contains conclusions and directions for future research in this area. Acknowledgements follow Section 6. References follow acknowledgements.

2. Theories and Concepts of Human–Robot Co-Manipulation

There is no fixed definition of human–robot co-manipulation of objects. However, to be recognized as human–robot co-manipulation, an object needs to be manipulated jointly by at least one human and one robot at a time in a task setup [18]. The object manipulation may include at least one of these functions: grasping an object, lifting an object, holding an object, carrying or transporting or moving an object, and placing or positioning an object [19,20,21]. There may be different forms of liaison between humans and robots for human–robot co-manipulation. For example, one human and one robot, one human and multiple robots, multiple humans and one robot, and multiple humans and multiple robots may co-manipulate a single object or multiple objects in a human–robot co-manipulation task [22,23].

The movement of a co-manipulated object is usually linear. However, the movement of the object may be rotational or a combination of linear and rotational in a human–robot co-manipulation task [24]. The human–robot co-manipulation may be along a single axis (known as single DOF co-manipulation) and multiple axes (known as multi-DOF co-manipulation) [25]. The autonomy levels between the human and the robot in a human–robot co-manipulation task may vary. However, the autonomy levels of the human and of the robot should be non-zero. If we can express the autonomy levels of the human and of the robot in percentage (%), then the total autonomy of the human and of the robot should be 100% [26]. Again, the role of the human and of the robot in a human–robot co-manipulation task should be complementary and thus their cooperation should be symbiotic [27]. In a human–robot co-manipulation task, other humans may take part in the task with different roles and capacities. For example, one or multiple humans may supervise, monitor, analyze or operate the human–robot co-manipulation system. However, these supporting or auxiliary humans should not be included in the human or humans collaborating with the robot or robots in actual manipulation of the object(s) [28]. The roles of collaborating and auxiliary/supporting humans should be identified separately.

Ideally, objects of any size, shapes, surface textures, materials, weights, etc., may be co-manipulated by humans and robots [29]. The objects may be both rigid (strong) and fragile. Solid objects are typically considered for human–robot co-manipulation in most cases. However, the types of objects for human–robot co-manipulation may depend on the actual needs of industries, and the structure of human–robot co-manipulation systems may be designed and adjusted according to object types [30]. Human–robot co-manipulation may use different robotic manipulation technologies. For example, the manipulation may be of power-assist type or nonpower-assist type [31]. Human–robot co-manipulation may be facilitated or augmented by different other technologies such as artificial intelligence, machine learning, control technologies and human factors [32]. However, such augmentation is not required to be a human–robot co-manipulation task.

3. Analysis of Related Work

The literature shows growing interest in developing suitable metrics and associated methods for assessing human–robot systems. The efforts were initiated at least two decades ago with the aim of evaluating human–robot interactions [33]. Saleh and Karray worked toward generalized performance metrics for human–robot interaction [34]. In [14], the authors conducted a survey regarding assessment metrics for interactions between humans and robots. An assessment scheme was proposed for assessing and analyzing cybersecurity performance in robots [13]. Marvel et al. came up with broader ideas of test methods and metrics for effective HRI in collaborative human–robot teams [15,35]. We, the author, initiated an effort toward developing necessary metrics for assessing an automotive manufacturing task performed in collaboration between a human and a robot [17]. The author also initiated developing an assessment scheme for human–robot co-manipulation of objects, as reported in [18,19].

In addition to the above efforts, different researchers focused on different aspects of assessments of human–robot systems, as follows. Figueredo et al. proposed metrics for human comfortability and manipulability during human–robot collaboration [36]. An evaluation approach for human–robot object co-manipulation under robot impedance control was proposed by Mujica, Benoussaad and Fourquet [37]. A holistic framework for robotic assessment on performance, software, and environmental adaptability was proposed by Muepu, Watanobe and Naruse [38]. Norton et al. proposed metrics for robot proficiency self-assessment in human–robot teams [39]. Basiri proposed performance evaluation methods for a domestic service robot regarding people perception, people following, and picking and placing [40]. Aller et al. investigated assessment metrics for humanoid robot locomotion [41], and Cao, Crandall and Goodrich investigated robot proficiency self-assessment [42]. Hümmer, Riedelbauch, and Henrich compared the consistency of user studies in human–robot systems [43]. Wan et al. proposed a performance and usability evaluation scheme for mobile manipulator teleoperation [44]. Zhang et al. derived a framework for assessing team performance of human–robot collaboration in construction [45]. Tan and Arai proposed a method of analytic evaluation of a human–robot system for collaboration in a cellular manufacturing system [46]. Inam et al. investigated risk assessment for human–robot collaboration in an automated warehouse scenario [47]. Schuster, Keebler, Zuniga and Jentsch investigated situation awareness measurement and performance in human–robot teaming [48]. Hoffman proposed the fluency evaluation method in human–robot collaboration [49], and so forth.

Review of this literature shows that only a few of the assessment efforts were directed toward the holistic or comprehensive assessment of human–robot collaborative systems [15,17,35]. Instead, most of the assessment efforts focused on some specific aspects of assessment of human–robot system [36,49]. Again, only a few of the assessment efforts were directly related to human–robot object co-manipulation [17,18,19,36,37,40]. The literature review shows only a few assessment initiatives that were taken toward deriving a generalized assessment framework of a human–robot system [17,18,19], though the work was not verified experimentally. Instead, most of the assessment schemes were specially developed for specific systems. Furthermore, efforts toward validating the assessment metrics and methods through comprehensive experimental evaluation were not observed in the literature. As a result, a comprehensive and generalized scheme comprising suitable criteria, metrics and methods for assessing human–robot object co-manipulation tasks as well as a dedicated effort toward validating practicality, generality, usability and effectiveness of the proposed scheme still demands serious attention, which we address herein.

4. Materials and Methods

4.1. Development of the Human–Robot Co-Manipulation System (The First System)

A 1DOF (up and down motions) human–robot co-manipulation system was developed, as Figure 1 shows. The co-manipulation system was a power assist robotic system (PARS) [12]. The robotic system was developed to be used to co-manipulate heavy objects (e.g., objects of 5–10 kg weights) in collaboration with a human co-worker. The object was tied to the lower end of the object holder. A force sensor was set up with a handle. A force sensor and an overhead laser position sensor were used to measure the human force applied to the PARS via the object and human hand movement during object co-manipulation, respectively. An encoder embedded in the manipulator motor was used to measure the kinematics of the PARS. A human co-worker could hold the handle and manipulate (vertically lift and lower down) the object in collaboration with the robotic system [12].

The equation of motion (EOM) for lifting an object with the PARS was derived as Equation (1).

$$m{\ddot{x}}_{de}+K_{lsm}\dot{x_{de}}+mg+F_{ff}=f_{hu}+f_{actuatorf}$$

(1)

In Equation (1), the parameters were defined as follows:

• $F_{ff}$: friction force (N);
• $f_{actuatorf}$: actuator force (N);
• $f_{hu}$: force applied by the human co-worker (i.e., load force) (N);
• g: acceleration due to gravity (m/s²);
• $K_{lsm}$: viscosity played role in the mating surface of the linear actuator (N.s.m.⁻²);
• m: virtual mass of the co-manipulated object (kg);
• x: actual displacement of the co-manipulated object (m);
• $x_{de}$: desired displacement of the co-manipulated object (m).

It was assumed that the human co-worker would manipulate the object naturally. It means that the human would not feel that the object was tied with a robotic system. Instead, the human would feel that they were manipulating the object as a free object [12]. Considering the targeted free or unrestricted motion dynamics, Equation (1) could be simplified to Equation (2). A feedback position control system was developed as illustrated in Figure 2 based on Equation (2) for the power assist robotic system for human–robot object co-manipulation. The aim of the feedback control was to enhance accuracy in the human–robot co-manipulation task, which could help determine the assessment metrics and methods more accurately [12].

$$m{\ddot{x}}_{de}+mg=f_{hu }$$

(2)

4.2. Experiment 1: Deriving Assessment Criteria, Metrics and Methods

4.2.1. Subjects

We recruited 33 human subjects (science and engineering students). Out of 33 subjects, 18 subjects were male and 15 were female. The mean age (as reported by the subjects) was 23.76 years (STD $=2.59)$. It was not a requirement for the recruited human subjects to have prior experiences with human–robot collaboration research. We then recruited 8 human subjects (graduate students, engineers) who had experience of conducting academic research in the areas of human–robot collaboration and co-manipulation. We called those human subjects “researchers”. We collected written consent from each subject for participating in the experiment. The subjects were unpaid for their participation.

4.2.2. Research Questions

We adopted the following two research questions (RQs) for co-manipulation of objects with the PARS demonstrated in Figure 1:

(i)

RQ 1: What are the appropriate criteria that can be used to assess performance of the human–robot co-manipulation task with the PARS? What should be the metric (way of expressing the assessment or measurement) of each criterion? How can we assess or measure each criterion (assessment method)?

(ii)

RQ 2: What are the criteria that truly reflect the effectiveness of interactions between a human subject and the robotic system for the co-manipulation task? What should be the metric (way of expressing the assessment or measurement) of each criterion? How can we assess or measure each criterion (assessment method)?

Answers to the above questions could help develop a set of criteria and metrics that could be used to assess and benchmark the performance and the effectiveness of human–robot interactions for the HRC manipulation system. It also could help us think of the method or strategy of assessing or measuring each criterion of the assessment metrics.

4.2.3. Experimental Procedures

We adopted the following research methods and procedures to derive suitable metrics and associated methods to assess performance and interactions for the human–robot collaborative manipulation system introduced in Figure 1 [13,14,15,16,17]:

• Literature review: We reviewed the literature related to human–robot object co-manipulation, e.g., [15,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49]. While conducting the literature review, we made a list of all the criteria that the researchers proposed or used to assess the performance and interactions between humans and robots for human–robot collaborative systems.
• Experimental experience: We conducted an experiment with the PARS shown in Figure 1. Each subject needed to separately collaborate with the robotic system (PARS) as it is illustrated in Figure 1 to co-manipulate an object tied with the PARS. At the end of the experiment, the subject was asked to answer the two research questions we adopted previously, and the responses of the subject were recorded.
• Expert opinions: Each researcher observed one trial of the human–robot co-manipulation of the object that the human subjects performed in collaboration with the PARS. Then, each researcher was asked to answer the two research questions we adopted previously, and the responses of the researcher were recorded.
• Analysis: We analyzed research results in human–robot co-manipulation that we obtained previously, e.g., [50,51,52], analyzed the human–robot co-manipulated task, illustrated in Figure 1, performed by each human subject, and made a list of all possible criteria that could be used to effectively assess the performance and human–robot interactions for the co-manipulated task.
• Synthesis: We then synthesized all the above information related to the assessment criteria, metrics and methods for the performance and human–robot interactions for co-manipulated tasks. The accumulated and synthesized information was further analyzed and categorized to derive a comprehensive set of assessment criteria, metrics and methods to assess the HRC performance and interactions between the human co-workers (subjects) and the robot for the HRC manipulation task [13,14,15,16,17]. Here, the assessment criterion meant what to assess or measure, the assessment metric meant how to express the measured or assessed criterion and the assessment method meant how to assess or measure each criterion. Each assessment criterion might have single or multiple assessment metrics and associated assessment methods.

When deciding criteria for human–robot co-manipulation performance, the criteria that could reflect key performance indicators (KPI) were prioritized [50,51,52]. Similarly, when deciding the criteria for human–robot interactions (HRI), the criteria that were practical and that could impact the KPI intensely were prioritized [52,53]. We emphasized objective assessment methods and metric for each criterion. However, we also proposed subjective assessment metrics and methods when objective metrics and methods caused difficulty in the assessment, or the objective assessment was not possible. Both objective and subjective criteria were proposed as they could be used to crosscheck the assessment results, which could enhance the reliability of the HRC metrology [33,53].

4.3. Experiment 1 Results and Analyses

Based on a thematic analysis using the information we collected through the experimental procedures as per the above [54], we identified the two themes of assessment criteria for human–robot co-manipulation tasks, as the following:

(i)

Criteria for assessments related to co-manipulation performance [55], and

(ii)

criteria for assessments related to human–robot interaction (HRI) [56,57].

We then split the HRI related assessment criteria into two categories or types, as follows:

(i)

Physical HRI (pHRI) criteria [56], and

(ii)

cognitive HRI (cHRI) criteria [57].

We then derived at least one metric for each assessment criterion. The assessment metric was either subjective or objective in nature depending on the criterion. We then proposed the most appropriate method to determine each metric considering the subjective or objective nature of the metric. Sometimes, the assessment methods were proposed as computational models comprising relevant parameters depending on the nature of the metrics and assessment criteria. For each subjective metric, we proposed both quantitative and qualitative assessment methods where possible for triangulation [58,59]. For qualitative assessments, we proposed that HRC manipulation activities should be observed and the key observational findings reflecting the overall quality of the HRC manipulation tasks should be noted down. For quantitative assessments, we tried to express the metric as values or quantities. The values or quantities could be determined subjectively using different rating scales, or the values could come from different sensors. For example, we proposed the usage of a Likert scale as the method, and the rating score as the metric to assess an HRI criterion where applicable. Here, the assessment criterion means “what to assess or measure”, the metric means “how to express the assessment or measurement results”, and the assessment method means “how to determine or obtain the assessment results” [54].

4.3.1. Assessment of Co-Manipulation Performance

Table 1. Proposed criteria, and associated metrics and methods for assessing co-manipulation performance [17].

Criteria	Metrics	Methods
Efficiency	Percentage (%)	${\displaystyle \frac{T_t}{T_m}}\times 100\%$
Accuracy	Percentage (%)	$(1-{\displaystyle \frac{{|P}_t-P_m|}{P_t}})\times 100\%$
Success rate	Percentage (%)	${\displaystyle \frac{T_s}{T_M}}\times 100\%$

shows the proposed metrology for assessing co-manipulation performance [55]. As Table 1 shows, the ratio between target time for manipulation (T_t) and measured time for manipulation (T_m) may be used to compute the efficiency of the co-manipulation task. A complete co-manipulation task may be divided into several segments. Efficiency may be computed for each segment or for the complete co-manipulation task. One can consider positional accuracy of the object co-manipulated by the human and the robot. For each co-manipulation task, there is a target location or position where the object should be positioned during co-manipulation. The target location or position can be expressed using the 6DOF coordinate system, which consists of 3 linear axes and 3 angular axes. Assume that $P_t$ is the value of the target position of the object along a linear axis of movement. If we determine the difference between $P_t$ and $P_m$ (actual position of the co-manipulated object along that linear axis of movement obtained through measurement) along that linear axis and express it in percentage then we can compute the (in)accuracy percentage of the co-manipulation task. As an example in Figure 3, P_m is the measured position and P_t is the target position along the X-axis for a rectangular object co-manipulated by the human and the robot jointly. In such a case, an equation (a computational model) as Table 1 shows was proposed to calculate the manipulation accuracy using those two parameters (P_m, P_t). The positions may be measured using a ruler or a position sensor (e.g., a laser sensor, a potentiometer, an ultrasonic sensor, etc.). The (in)accuracy can also be measured along the Y-axis and the Z-axis (the vertical axis) following the same formula (computational model). The angular positional (in)accuracy can also be measured following the same formula by considering the measured and target angular positions (determined using angular position sensors) with respect to each of the 3 different axes of rotation separately [17]. The (in)accuracy measures are independent of target positions as the measures are expressed in percentages.

As shown in Table 1, one can use two parameters to compute the success rate in human–robot object co-manipulation: (i) counts of total manipulation events in a task cycle denoted by T_M and (ii) counts of successful manipulation events in that task cycle denoted by T_s. The empirical formula shown in Table 1 can be used to compute the success rate based on these two parameters. One can manually count these two parameters based on observation of the co-manipulation task, or the task can be videotaped, and the values of the parameters can be extracted from videos, or computer vision methods can be used to obtain the parameters [60].

4.3.2. Assessment of pHRI

The proposed pHRI metrology for the human–robot co-manipulation task is shown in

Table 2. Proposed criteria, and associated metrics and methods for assessing pHRI [17].

Criteria	Metrics	Methods
Maneuverability	Rating scores of subjective assessments	A 7-point or a 5-point Likert scale can be used to assess maneuverability subjectively for each subject that participates in human–robot co-manipulation tasks.
Motion	Rating scores of subjective assessments, or magnitudes and fluctuations in co-manipulation accelerations	A 7-point or a 5-point Likert scale can be used to assess motion subjectively for each subject that participates in human–robot co-manipulation tasks. Different types of position measuring instruments (position sensors) can be used to capture or measure magnitudes and fluctuations in co-manipulation accelerations.
Naturalness	Rating scores of subjective assessments	A 7-point or a 5-point Likert scale can be used to assess naturalness subjectively for each subject that participates in human–robot co-manipulation tasks.
Ease of work	Rating scores of subjective assessments	A 7-point or a 5-point Likert scale can be used to assess ease of work subjectively for each subject that participates in human–robot co-manipulation tasks.
Stability	Rating scores of subjective assessments or magnitudes and fluctuations in vibration during a co-manipulation task or number of sudden inactivity of the co-manipulation system during a co-manipulation task	A 7-point or a 5-point Likert scale can be used to assess stability subjectively for each subject that participates in human–robot co-manipulation tasks. Measurements of magnitudes and fluctuations in vibration during human–robot co-manipulation. Number of inactivity of human–robot co-manipulation system (counted during the co-manipulating task).
Safety	Rating scores of subjective assessments or number of accidents, injuries, and damage or percentage (%) of risk or level of risk	A 7-point or a 5-point Likert scale can be used to assess safety subjectively for each subject that participates in human–robot co-manipulation tasks. Number of accidents, injuries or damage during human–robot co-manipulation. Formal assessment of risk can be conducted to assess percentage (%) of risk, or the level of risk during human–robot co-manipulation.
Engagement	Rating scores of subjective assessments or percentage of time the human is physically connected or attuned with the task	A 7-point or a 5-point Likert scale can be used to assess engagement subjectively for each subject that participates in human–robot co-manipulation tasks. Work sampling method to compute percentage (%) of time a human co-worker can keep them engaged with the human–robot co-manipulation task physically or attuned with the task bodily.
Fluency	Percentage (%) of time the human and the robot are found working (none of them are idle)	Stopwatches. Recording co-manipulation displacement profiles or video recording co-manipulation tasks to identify the time to determine percentage (%) of time the human and the robot are found working (none of them are idle).
Symbiosis	Rating scores of subjective assessments or the ratio of sub-activities	A 7-point or a 5-point Likert scale can be used to assess symbiosis subjectively for each subject that participates in human–robot co-manipulation tasks. The number of sub-activities during human–robot co-manipulation tasks where a robot needs to depend on a human and the number of sub-activities during human–robot co-manipulation tasks where a human needs to depend on a robot can be determined, and the ratio between the numbers can be determined.
Autonomy	Percentage (%) in decision making or autonomy ratio	The number of sub-activities during human–robot co-manipulation tasks where a robot needs to make decisions and the number of sub-activities during human–robot co-manipulation tasks where a human needs to make decisions can be determined, and the ratio between the numbers can be determined.

[56]. Human’s haptic feeling or perception (i.e., perceived heaviness, kinesthetic and tactile perceptions and proprioception) during the co-manipulation task may be termed as maneuverability. Whether object displacement, velocity and acceleration are low or high compared to the expectation of the human subject or the co-worker may mean perceived motion of the co-manipulation task. Magnitudes and fluctuations in acceleration profiles during object co-manipulation, as illustrated in Figure 4, may help assess appropriateness of the motion objectively [12].

Naturalness may mean whether a human co-worker feels no difference between when they manipulate an object manually and when they manipulate the same object tied to another mechanical device. In the proposed task scenario, it may mean whether or to what extent a human co-worker feels that they manipulate a free object (not tied to a mechanical device) even though the object is held by the robot (another device). With respect to the proposed task scenario, ease of work may mean how easily a human co-worker can manipulate an object in collaboration with the robot. Naturalness and ease of work may be interrelated, but they carry different meanings as per the above. Stability can be realized through the presence or absence of oscillations or vibrations, sudden inactivity of the system, etc., during HRC manipulation, and the effects of these phenomena on object manipulation, system structure and configuration, work environment, etc. The magnitude and fluctuation measurements of vibrations, or the count of inactive states of the co-manipulation system may also be used to assess stability of the co-manipulation system. Table 2 shows different metrics and associated methods to assess safety in the human–robot co-manipulation task [47].

The term engagement may mean how a robot and a human are bodily or physically tied to each other during the co-manipulation task, and the work sampling method can be used to assess engagement [61]. The term team fluency may mean the percentage of time the robot and the human do not remain idle [49]. The term symbiosis may mean how the robot and the human depend on each other during the co-manipulation task [62]. The nature and extent of the symbiosis can determine the success of collaborative manipulation. The symbiosis may help understand the strengths and weaknesses of the human and of the robot so a complementary task allocation between them can be implemented. The term autonomy may mean the authority or ability of the robot or of the human to make necessary decision(s) during the co-manipulation task [63].

4.3.3. Assessment of cHRI

The proposed cHRI metrology for HRC manipulation tasks is shown in

Table 3. Proposed criteria, and associated metrics and methods for assessing cHRI [17].

Criteria	Metrics	Methods
Overall cognitive workload or cognitive workload due to mental demand, physical demand, temporal demand, performance, frustration and effort	Subjective score between 0% and 100%	NASA TLX (paper-based or software-based).
Trust	Subjective rating score or computed values of robot trust in human and human trust in robot between 0 (or 0%) and 1 (or 100%)	A 7-point or a 5-point Likert scale can be used to assess trust subjectively for each subject that participates in human–robot co-manipulation tasks. Computational models can be used for computing robot trust in humans and human trust in robots between 0 (or 0%) and 1 (or 100%) ignoring over trust, mistrust, distrust and under trust [8].
Situation awareness	Subjective rating score or test score	A 7-point or a 5-point Likert scale can be used to assess situation awareness subjectively for each subject that participates in human–robot co-manipulation tasks. SAGAT (situation awareness global assessment test) scores.
Long-term relationship	Rating scores of subjective assessments	A 7-point or a 5-point Likert scale can be used to assess long-term relationship subjectively for each subject that participates in human–robot co-manipulation tasks.

[57]. As Table 3 shows, the cognitive workload is the load imposed on the cognitive processes of the human subjects during the HRC manipulation task. The human subject’s cognitive workload can be expressed in six interrelated elements or dimensions such as the mental demand, physical demand, temporal demand, performance demand, frustration caused, and the effort required to perform the manipulation task. The well-known NASA TLX can be used to measure the cognitive workload [64]. Human trust in the robot can be assessed subjectively using a Likert scale or a computational model can be derived to compute the artificial trust of the human in the robot [8,65]. We may also consider human–robot bidirectional trust. In such a case, we can derive a model to compute human trust in robots and another model to compute robot trust in humans, though both trusts are artificial trust [8]. Situation awareness is the human subject’s ability to perceive the work environment accurately and promptly while working with a robot in the co-manipulation task [48]. The human subject can assess their situation awareness subjectively using a 7 or 5-point Likert scale. Another method called the SAGAT (situation awareness global assessment test) can also be conducted to assess situation awareness quantitatively [66]. The long-term relationship is the prospect and possibility that a human co-worker may be accustomed to working with the robot and feel comfortable to continue working with the robot in the future. It may also mean whether a human co-worker considers the robot as a team partner instead of considering it as a machine for the manipulation task. The human subject can assess their feelings of potential long-term relationship with the robot partner subjectively using a Likert scale [8].

4.4. Experiment 2: Validation of Assessment Criteria, Metrics and Methods

4.4.1. Development of the Human–Robot Co-Manipulation System (The Second System)

In addition to the power assist robotic system demonstrated in Figure 1, we developed a human–robot hybrid cell using a robotic manipulator where a human co-worker and the robotic manipulator collaborated on picking, assembling and placing a center console module of a car body, as demonstrated in Figure 5 [67]. In the beginning of the collaboration, the large center console frame was placed on the table on the left side of the human, and two other small components were kept on the table on the right side of the human. The robot picked up the small components and placed those components on the table surface in front of the human following a predefined task sequence. The human then assembled the small components with the large center console frame on the table surface in front of them following task sequence and instructions displayed on the computer screen placed in front of the human, and then the robot and the human collaboratively manipulated (co-manipulated) the assembled (finished) center console module from the table center to the right side of the human and placed it there [8]. In the demonstrated collaborative task scenario in Figure 5, the robot manipulated the small parts independently, and the human assembled the parts with the center console frame independently. The co-manipulation operation occurred when the assembled center console module was dispatched to the right side of the table [8]. The scenario demonstrated in Figure 5 may represent human–robot co-manipulation tasks observed in real-world collaborative assembly operations in industries.

4.4.2. Subjects

In total, 32 subjects (17 males, 15 females; mean age 22.89 years, STD = 3.64) were recruited. The subjects were undergraduate and graduate students and researchers. Those subjects were different from the subjects that participated in Experiment 1. Each subject’s consent to participate in the experiment was collected separately. The subjects received brief instructions and training before they participated in the experiment.

4.4.3. Experimental Procedures

There were two phases of this experiment (Experiment 2). In the first phase of the experiment, each subject introduced in Section 4.2.1 of Experiment 1 separately collaborated with the power assist robotic system (PARS) and co-manipulated the object tied to the power assist robotic system, as demonstrated in Figure 1. For each subject for each trial, the HRC manipulation performance, pHRI and cHRI were assessed (necessary data were collected) following the criteria, methods and metrics proposed in Table 1, Table 2 and Table 3, respectively. In the second phase of the experiment, each subject introduced in Section 4.4.2 separately collaborated with the robot manipulator demonstrated in Figure 5 to perform the assembly and dispatch operations of the center console module of the car body. For each subject for each trial, the HRC manipulation performance, pHRI and cHRI were assessed (necessary data were collected) following the criteria, methods and metrics proposed in Table 1, Table 2 and Table 3, respectively. In each phase, each subject was instructed on the assessment scheme (criteria, metrics, methods) before they participated in the experiment. In each phase, each subject needed to respond to a set of usability assessment questionnaires following a 7-point Likert scale at the end of the co-manipulation trial to assess the usability of the proposed assessment scheme [8].

4.5. Experiment 2 Results and Analyses

We determined the mean result for each assessment criterion in Table 1, Table 2 and Table 3 for the two different experimental phases of Experiment 2 (two different types of co-manipulation experiments using the power assist robotic system and the robotic manipulator respectively). The results are presented in

Table 4. Mean HRC manipulation performance assessment results for the co-manipulation tasks for the two different collaborative robotic systems.

Criteria	Metric	Mean Values in Percentage (Standard Deviation)
		PARS	Robot Manipulator
Efficiency	Percentage (%)	99.04 (2.11)	98.23 (3.18)
Accuracy	Percentage (%)	98.69 (2.38)	97.51 (3.74)
Success rate	Percentage (%)	100 (0)	100 (0)

,

Table 5. Mean pHRI assessment results for the co-manipulation tasks for the two different collaborative robotic systems.

Criteria	Metric	Mean Values (Standard Deviation)
		PARS	Robot Manipulator
Maneuverability	Subjective rating score (7-point Likert scale between 1 and 7)	6.77 (0.13)	6.68 (0.17)
Motion	Subjective rating score (7-point Likert scale between 1 and 7)	6.72 (0.10)	6.63 (0.14)
	Fluctuations in manipulation acceleration (%)	8.08 (0.19)	8.23 (0.26)
Naturalness	Subjective rating score (7-point Likert scale between 1 and 7)	6.58 (0.20)	6.49 (0.22)
Ease of work	Subjective rating score (7-point Likert scale between 1 and 7)	6.82 (0.21)	6.74 (0.16)
Stability	Subjective rating score (7-point Likert scale between 1 and 7)	6.29 (0.18)	6.57 (0.09)
	Fluctuations of vibrations (changes in vibration amplitudes) during HRC manipulation (%)	5.01 (0.07)	5.06 (0.04)
	Number of inactivity of human–robot co-manipulation system (counted during the co-manipulating task)	0	0
Safety	7-point Likert scale rating scores (subjective assessment scores)	6.84 (0.16)	6.92 (0.24)
	Number of damages, accidents, injuries occurred during human–robot co-manipulation task	0	0
	Percentage of risk (%) occurred during human–robot co-manipulation	0	0
	Level of risk in human–robot co-manipulation	Low	Low
Engagement	7-point Likert scale rating scores (subjective assessment scores)	6.96 (0.15)	6.91 (0.13)
	Percentage (%) of time a human co-worker could keep them engaged with the human–robot co-manipulation task physically or attuned with the task bodily.	100 (0)	100 (0)
Fluency	Percentage (%) of time the human and the robot were found working (none of them were idle)	100 (0)	100 (0)
Symbiosis	Subjective rating score (7-point Likert scale between 1 and 7)	6.93 (0.06)	6.46 (0.12)
	Sub-activity ratio	1:1	1:1
Autonomy	Percentages (%) in decision making (human %, robot %)	50, 50	50, 50
	Decision making (autonomy) ratio	1:1	1:1

and

Table 6. Mean cHRI assessment results for co-manipulation tasks for the two different collaborative robotic systems.

Criteria	Metric	Mean Values (Standard Deviation)
		PARS	Robot Manipulator
Cognitive workload due to mental demand	Subjective score between 0% and 100%	20.64 (1.12)	19.47 (1.29)
Cognitive workload due to physical demand		10.19 (1.23)	13.14 (1.08)
Cognitive workload due to temporal demand		16.17 (1.03)	21.03 (2.42)
Cognitive workload due to performance		19.26 (1.41)	16.84 (1.73)
Cognitive workload due to frustration		11.44 (1.29)	12.98 (1.66)
Cognitive workload due to effort		9.62 (1.14)	14.53 (1.87)
Overall cognitive workload		17.28 (1.20)	16.33 (1.68)
Trust	Subjective rating score (7-point Likert scale between 1 and 7)	6.76 (0.28)	6.64 (0.24)
	Computed human trust in the robot and robot trust in the human between 0 and 1 [8]	0.96 (0.04), 0.0.97 (0.18)	0.98 (0.06), 0.0.94 (0.11)
Situation awareness	Subjective rating score (7-point Likert scale between 1 and 7)	6.92 (0.24)	6.86 (0.31)
	SAGAT test score (%)	100 (0)	100 (0)
Long-term relationship	Subjective rating score (7-point Likert scale between 1 and 7)	6.42 (0.17)	6.39 (0.10)

as follows. The results showed that it was possible to successfully assess the co-manipulation tasks in two different robotic systems for all the assessment criteria identified in Table 1, Table 2 and Table 3 following the assessment methods proposed in Table 1, Table 2 and Table 3 and it was possible to successfully express the assessment results through the metrics derived in Table 1, Table 2 and Table 3. Thus, the results clearly showed the effectiveness of the proposed assessment scheme (HRC metrology). The results also helped us compare co-manipulation performance, pHRI and cHRI between the two co-manipulation systems based on the same assessment scheme, which helped toward justifying the benchmarking significance of the assessment scheme [17].

We determined the mean assessment value with standard deviation for each usability assessment criterion to assess the usability of the derived assessment scheme (assessment criteria, methods and metrics) for both robotic co-manipulation systems separately. Figure 6 shows the results for the usability of the assessment scheme for the two robotic platforms separately. The results showed that the subjects perceived that the assessment criteria, methods and metrics were highly usable and useful for both robotic platforms. The results thus proved the effectiveness, usefulness, and validity of the proposed assessment criteria, methods and metrics, which we termed here as HRC metrology [17]. The subjects gained experience with the co-manipulation systems when they used the co-manipulation systems for experiments, and they were also aware of the entire assessment scheme and its objectives. As a result, justification of usability of the assessment scheme based on the opinions of the subjects should be rational. Furthermore, a portion of the subjects were HRC system researchers, and their opinions further strengthened the justification of the assessment scheme. However, verification of the assessment scheme with more experienced HRC researchers and industry professionals could make the justification more rational and reliable [17].

5. Discussion

5.1. Strengths of the Proposed Assessment Scheme

Strengths of the proposed assessment scheme are that the scheme is comprehensive in nature because the scheme consists of objective, subjective, quantitative and qualitative criteria, metrics and methods [8]. We accumulated almost all relevant assessment criteria for human–robot co-manipulation tasks and included them in the scheme, which enhanced the scope of the scheme [35]. In many cases, we proposed more than one metric and method for assessing a single criterion, which also enhanced the scope, flexibility and comprehensiveness of the scheme. The scheme is practical, and its usability is justified, which are the strong sides of the scheme [17]. The comprehensive and experimentally verified assessment scheme removed or reduced the limitations of the past efforts in [17,18,19].

5.2. Limitations of the Proposed Assessment Scheme

The main limitations of the proposed scheme are: (i) we used limited subjects in the experiments (though it might be justified as power analysis was conducted to determine the minimum number of subjects) [68], (ii) many of the subjects did not possess enough experience in human–robot collaborative research, (iii) we did not verify the proposed metrics and methods with end users (industry workers), (iv) we verified the scheme with only two different robotic platforms, but it was not verified whether the proposed criteria, metrics and methods would work for all other types of HRC manipulation tasks developed with other types of collaborative robotic platforms and task requirements [69], (v) more assessment criteria could be incorporated in the scheme, (vi) we focused more on subjective criteria than on sensor-based objective criteria though the scheme consisted of both subjective and objective criteria [70,71,72,73], etc.

5.3. Weighing Scheme

It was possible to determine the weight of each criterion (relative importance of a criterion with respect to others) based on expert observers’ numerical ratings and conduct analysis on inter-rater reliability and bias mitigation [74]. Determination of weights was also possible for assessment metrics and methods where multiple metrics and methods were proposed. In such a case, it could be useful to determine influential criteria, metrics and methods, and the guidance of selection of criteria, metrics and methods for specific applications could be provided. However, weights were not determined. Having no weight means that all criteria, metrics and methods were equally important and useful for successful application of the assessment scheme. Nonetheless, it is possible to determine the weights and add them to the top of the results presented herein in the next stage of the research [75].

5.4. Significance of the Results from the Perspective of Co-Manipulation Theories

We discussed relevant theories and concepts related to human–robot co-manipulation in Section 2, which defined the scope of human–robot co-manipulation tasks. The robotic platforms we presented in this article truly illustrated human–robot co-manipulation tasks, where grasping, lifting, holding, transporting and placing or positioning an object could be clearly identified in both robotic setups [19,20,21]. The PARS was a 1DOF system, but the robot manipulator was a 6DOF system. In both cases, only one human collaborated with the robot, and they co-manipulated only one object at a time. One robotic system provided power assistance, the other did not [31]. In the PARS, the movement was linear, but the movement was non-linear for the robotic manipulator. The roles of collaborating humans (human subjects) and auxiliary humans (e.g., the experimenter, observers) were identified separately in the experiments. Ideally, objects of any size, shapes, surface textures, materials, weights, etc., can be co-manipulated by humans and robots in human–robot co-manipulation [29]. The objects (parts) can be both rigid (strong) and fragile. In our cases, both robotic systems were used to co-manipulate irregular shaped solid objects/parts with different surface textures [30]. Human–robot co-manipulation may be facilitated or augmented by different other technologies such as artificial intelligence, machine learning, control technologies, human factors, etc. [32,76,77,78]. We implemented a simple control scheme for the PARS, but no control was applied to the robotic manipulator [79,80].

Thus, we see that the results we obtained for the two robotic platforms addressed many of the criteria of the theories and concepts introduced in Section 2. However, the presented research setups and results do not address all the relevant theories and concepts related to human–robot co-manipulation we discussed in Section 2. This is because one or two selected robotic platforms cannot cover the features of all the theories and concepts related to human–robot co-manipulation tasks. Instead, the two robotic platforms we used in this article simply represented all possible human–robot co-manipulation systems and associated theories and concepts.

5.5. Verification, Validation, Generalization and Standardization of the Results

Verification is a process and validation is the decision and conclusion made based on the outcomes of the verification process. By ‘verification’ we mean it was tried to check if the assessment scheme was useful, feasible and adequate. Once it was verified and once the verification results were favorable (Figure 6), it was validated [81]. However, the validation of the proposed assessment scheme was done amid limited conditions, which could be further strengthened if the assessment scheme was verified using more experimental platforms with diversified task scenarios. The term generalization means making the assessment scheme appropriate for all human–robot co-manipulation tasks in all conditions. While this should be the target of this research, generalization was attempted herein with limited capacity. The assessment scheme proved useful, feasible and adequate for two different experimental platforms. As a result, the scheme was somewhat generalized, which could be further stretched by verifying the assessment scheme using more experimental platforms with various task scenarios.

In this paper, the objective was not to determine standards of assessment methods and results (numerical values) for human–robot co-manipulation tasks [82]. Instead, the objective was to derive a comprehensive set of assessment criteria and related assessment metrics and methods that could be used to assess human–robot co-manipulation tasks conveniently and adequately. To standardize the assessment methods, SOPs (standard operating procedures) for each assessment method would need to be developed or such SOPs developed by national or international institutions could be used. Similarly, standards for the assessment results would need to be developed or standards developed by national or international institutions could be used to evaluate and benchmark human–robot co-manipulation systems [82].

5.6. Similarities and Differences Between the Two Robotic Platforms

Both robotic platforms clearly demonstrated the features of human–robot co-manipulation [19,20,21,29,30,31,32]. Nonetheless, the PARS was a 1DOF system, but the robotic manipulator was a 6DOF system. The PARS provided linear manipulation, but the robotic manipulator provided non-linear manipulation. The object co-manipulated with the PARS was heavier than that co-manipulated with the robotic manipulator. The PARS was enriched with control technology, but the robotic manipulator was not. Finally, the PARS provided power assistance, but the robotic manipulator did not provide such power assistance. All these differences may impact the advantages, disadvantages, usability and performance assessment results of each robotic platform itself, but the differences should not impact the effectiveness of the assessment scheme since the scheme was proven effective for both robotic platforms [17].

5.7. Possibility of Adding More or New Assessment Criteria

Undoubtedly, we accumulated almost all relevant criteria for assessing human–robot co-manipulation tasks, which made the criteria comprehensive [17,18,19]. Nonetheless, it is still possible to augment the assessment criteria, assessment metrics and assessment methods of the proposed assessment scheme. To do so, we may need to repeat the procedures described in Section 4.2.3 for many other types of robotic platforms and human–robot co-manipulation tasks of many types. The human co-workers and observers selected for experiments may be larger in number and background.

5.8. Impact of Advanced Control and AI on the Results

Sometimes, advanced control methods and AI algorithms such as machine learning algorithms may be integrated into human–robot co-manipulation systems [32,76,77,78]. Such augmentation can impact the performance and HRI of the human–robot co-manipulation systems. However, the inclusion of control and AI algorithms may have no impact on the suitability and effectiveness of the proposed assessment scheme since the inclusion of control strategies and AI algorithms does not necessarily impact the assessment criteria, metrics and methods [51].

5.9. Applications and Advantages of the Results

The results can be proven very useful to assess different human–robot object co-manipulation systems using a single scheme or a single basis of evaluation as we presented herein. Such advantages can help human–robot co-manipulation designers, researchers, and practitioners evaluate the performance of a particular human–robot co-manipulation system and compare it with state-of-the-art systems in terms of their performance and interactional effectiveness. Thus, the scheme can help industry practitioners make purchase and deployment decisions of human–robot co-manipulation systems [35].

5.10. Scaling up the Systems for Industrial Settings

HRC systems we used in the experiments were laboratory systems. However, it is possible to upgrade the systems to meet industry requirements by simply scaling up the systems. For example, the PARS can be made by a multi-DOF system to enhance its capabilities to manipulate objects along different axes. Structures of both systems can be made stronger to make the systems capable of carrying larger and heavier objects. Conveyor belts can be added to robotic systems to input and output/dispatch objects easily and continuously. Multiple human workers can work with a single robot or multiple robots for different types of objects, which can enhance co-manipulation scope and productivity. To do so, the task setup should be scaled up, and the robot gripping mechanisms should be upgraded. It is also possible to connect the robotic system to the central control system of an industry through PLC [83].

6. Conclusions and Future Work

A 1DOF PARS was developed for co-manipulation of an object through collaboration between a robot and its human counterpart. Based on an experiment, literature reviews, our experiences with similar collaborative manipulation tasks and interviews with the subjects and relevant researchers, an HRC metrology was derived, which included purposefully developed criteria, metrics and methods that could be used to assess human–robot co-manipulation tasks. All the criteria proposed to assess human–robot co-manipulation tasks were divided into two assessment themes: (i) criteria to assess the performance of co-manipulation tasks, and (ii) criteria to assess interactions of humans with robots (HRI or human–robot interactions). The interactions of humans with robots (HRI) theme was further divided into two categories: (i) pHRI (physical human–robot interactions) indicating physical interactions between humans and robots, and (ii) cHRI (cognitive human–robot interactions) indicating cognitive interactions between humans and robots. Then, various metrics associated with practical methods were proposed to assess each category of criteria. The proposed metrics and methods were quantitative, qualitative, objective and subjective in nature, which made the metrology comprehensive and provided opportunities to crosscheck the assessment results. In another study, another robot–human hybrid cell was developed using a human and a robot, where the human collaborated with a robot manipulator to perform co-manipulation of an object (an automobile center console) in a human–robot collaborative assembly setup. Then, performance and HRI of both robotic co-manipulation systems were evaluated following the proposed assessment criteria, metrics and methods in a separate validation experiment. The results proved the assessment scheme effective, practical and usable. We also discussed the strengths and limitations of the proposed assessment metrics and methods. The proposed scheme seems to be better and more innovative in terms of comprehensiveness, scope and practicality than the state-of-the-art schemes. The proposed HRC metrology can be used to assess human–robot co-manipulation tasks for various real-world applications such as industrial manufacturing and manipulation operations, civil construction operations, disaster management, timber processing, logistics and transport activities, etc.

As our future tasks, we will expand the scheme by deriving more assessment criteria and their assessment metrics and methods. We will determine the weight of each assessment criterion, metric and method, determine a set of influential criteria and develop guidance of selection of influential criteria. We will also verify the effectiveness of the proposed assessment scheme using industry professionals in industry settings for different robotic co-manipulation tasks and robotic platforms. We will develop standards for the assessment methods by developing SOPs (standard operating procedures) for each assessment method and develop standards for assessment results for each assessment criterion. The standardized assessment scheme can be used to benchmark HRC performance and interactions in human–robot co-manipulation tasks.