Next Article in Journal
Enhancing Tool Wear Prediction Accuracy Using Walsh–Hadamard Transform, DCGAN and Dragonfly Algorithm-Based Feature Selection
Previous Article in Journal
Learning-Based Approaches to Current Identification from Magnetic Sensors
Previous Article in Special Issue
Application of Neural Network in Predicting H2S from an Acid Gas Removal Unit (AGRU) with Different Compositions of Solvents
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 23
Issue 8
10.3390/s23083826
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Implementation of a Hybrid Intelligence System Enabling the Effectiveness Assessment of Interaction Channels Use in HMI

by
Arkadiusz Gardecki
1,2,*,
Joanna Rut
3,*,
Bartlomiej Klin
1,2,
Michal Podpora
1,2 and
Ryszard Beniak
1,2
1
Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland
2
Weegree Sp. z o.o. S.K., 45-018 Opole, Poland
3
Faculty of Production Engineering and Logistics, Opole University of Technology, 45-272 Opole, Poland
*
Authors to whom correspondence should be addressed.
Sensors 2023, 23(8), 3826; https://doi.org/10.3390/s23083826
Submission received: 10 February 2023 / Revised: 31 March 2023 / Accepted: 6 April 2023 / Published: 8 April 2023
(This article belongs to the Special Issue Intelligent Sensors and Machine Learning)
Browse Figures
Versions Notes

Abstract

:
The article presents a novel idea of Interaction Quality Sensor (IQS), introduced in the complete solution of Hybrid INTelligence (HINT) architecture for intelligent control systems. The proposed system is designed to use and prioritize multiple information channels (speech, images, videos) in order to optimize the information flow efficiency of interaction in HMI systems. The proposed architecture is implemented and validated in a real-world application of training unskilled workers—new employees (with lower competencies and/or a language barrier). With the help of the HINT system, the man–machine communication information channels are deliberately chosen based on IQS readouts to enable an untrained, inexperienced, foreign employee candidate to become a good worker, while not requiring the presence of either an interpreter or an expert during training. The proposed implementation is in line with the labor market trend, which displays significant fluctuations. The HINT system is designed to activate human resources and support organizations/enterprises in the effective assimilation of employees to the tasks performed on the production assembly line. The market need of solving this noticeable problem was caused by a large migration of employees within (and between) enterprises. The research results presented in the work show significant benefits of the methods used, while supporting multilingualism and optimizing the preselection of information channels.

1. Introduction

In the developing world of advanced technologies and constantly changing working environment conditions, where time and quality of performed tasks and manufactured products are especially valued, enterprises are forced to search for new means of competitiveness. The advancement is made through the use of new technologies, created for the needs of economic and flexible production while affecting the quality of work performed also by newly hired employees with short work experience. Technical and organizational solutions supported by new technologies created for the implementation of production activities, the application of which is constantly being expanded, are a key element in the development of enterprises that must adapt to the requirements of Industry 4.0 and in the near future also to Industry 5.0.
The ever-changing reality and the evolving digitization of the industry force enterprises to transform toward the inclusion of new and visionary approaches [1,2]. New approaches imply in turn opportunities for enterprises to improve the quality of work organization and/or the selection of techniques that allow maintaining the continuity of the supply chain, and thus, increase the competitiveness of a company [3,4].
One of the foundations of Industry 5.0 is the co-existence and collaboration of men and machines in a natural environment. Controlling and managing the man–machine collaboration process require sufficient methods and ways to be utilized. The authors propose a novel Interaction Quality Sensor (IQS) integrated into the broader system fulfilling the instruction supporting the HMI role in the production line environment. This collaboration is expected to take place using natural communication, instead of specialized interfaces or protocols, shifting the focus towards human-oriented, context-aware, adaptive systems [5]. With the progress of Artificial Intelligence (AI), as well as NLP (Natural Language Processing), which is clearly seen in modern conversational {systems [6,7], the man–machine collaboration becomes an urgent and promising trend in modern industry. Nevertheless, despite recent significant advancements [8,9,10,11] in the field of Human–Computer Interaction (HCI) (namely: the Amazon Echo available since 2015 [12,13,14], Microsoft Cortana massively expanding in 2015 to numerous platforms [15], Google Speech announced in 2016, upgraded in 2018 [16] and 2022 [17], Google Assistant announced in 2016 [18,19], Google Nest introduced as Google Home also in 2016 [20,21,22,23,24], and the Apple Siri updated in 2017 [25,26]) and undoubtful progress in Artificial Intelligence and Machine Learning [27,28,29,30], conversational AI systems usually still tend to disappoint rather than to amaze their interlocutors (and there are some specific reasons for that, already diagnosed and described [31]). In the case of industrial solutions, an appropriate sensor network with insight into every aspect of the man–machine communication with regard to its purpose is required (allowing later to provide insight into the quality and efficiency of every stage of the process for a given worker/trainee). In many cases, the introduction of new tools and technologies into an enterprise ecosystem is associated with the need for a change, including expanding the scope of duties of employees, changing their job position, or opening new job positions focused on knowledge and sustainability [32]. However, intelligent digitization is not about replacing employees with robots or other automatic systems, but about increasing operational efficiency and work comfort by leveraging the importance of flawless and natural man–machine interaction [33]. Appropriately selected tools and IT technology ensure the improvement of the quality of employee workflows, increase the efficiency of the work carried out, improve communication and information flow, and support the development of the enterprise [34,35].
Currently, an increasingly popular way to optimize the work cycle in enterprises and production plants is the use of new technologies and IT tools to support company employees in their daily duties through the use of automation and robotization solutions. Robotics makes it possible to transfer repetitive activities from human employees to robots while allowing to shorten the overall time of process implementation and reducing costs and the number of production defects [36]. Enterprises that want to maintain a good position in the market and gain new contractors must strengthen their image, especially with regard to future prospects [37]. The use of robots increases the production capacity of enterprises, making them attractive partners in a competitive market. The possibilities of using robots in enterprises are even wider, namely, they are currently extremely useful in selected application fields, among others, in quality control processes [38]. The introduction of robots in enterprises contributes to numerous significant benefits, including improved customer service quality, greater motivation of employees to perform their work, as well as a sense of effectively performed work and its optimization [39]. The lifecycle of an enterprise assumes constant evolution toward positive financial income. Nowadays, when the world struggles with pollution and energy price spikes, the life cycle also includes lowering power consumption in the production process. Introducing new technologies such as robotics and AI positively impacts green innovation [40,41].
Despite the popularity of the use of new technologies and the current degree of development of robots, the problem of recruiting qualified employees is still an issue in many enterprises [42]. Many companies and organizations are currently facing difficulties in finding employees with the skills currently in demand [43,44]. Difficulties in acquiring qualified employees may partially be a consequence of the competency mismatch of candidates on the market with the dynamically advancing technological capabilities, digitization, and the evolving specificity and organization of many enterprises [45]. Moreover, the labor market dynamics forces companies to be able to quickly train employees to adapt to production and/or employment changes [46,47], and furthermore, the migration crisis [48,49,50], results in a new requirement—to be able to create workplaces for foreign-speaking employees [51]. All this causes the search for solutions that may support the functioning of enterprises without causing downtime in the execution of tasks. These challenges are met by new technology that offers support and solutions. The current pace of enterprise adoption of new technologies, artificial intelligence, robotics, and automation in the workplace is strongly accelerating [52]. The whole range of opportunities related to new technologies is changing. Artificial Intelligence, as well as robotics, open up new opportunities for many areas of business operations [53], including the HR department [54,55,56,57].
Mitigating human resource shortages requires the entrepreneur to make difficult decisions about delegating qualified employees to train new workers, whose integration with the workplace environment and processes can itself become a long-term task, which carries the risk of failure [58]. The training abilities of the senior staff directly affect not only the success rate achieved, but also the newly established relationships with new employees, and the level of adaptation of training to the skills and knowledge of a particular person [59,60]. The number of quantified parameters affecting successful training in the above case increases with the complexity of the activities performed at the workplace and the possible level of knowledge absorption by a person per unit of time. The unavailability of communication in the employee’s native language (and insufficient adjustment of training methods and materials) is a serious barrier that is difficult to eliminate [61].
In the case of a classic approach (employee training without involving advanced IT systems) to the management of the employee training process, the evaluation (assessing the effectiveness of the teaching-learning process) and the prediction of their achieved proficiency level are difficult to implement (mostly due to the lack of appropriate procedures implementations, as well as reliable data) [62]. Nevertheless, new technologies based on Machine Learning [57], predictive analytics [63], and Big Data [56] can significantly improve employee onboarding processes within an organization [64,65]. Moreover, they can also provide insight into the KPIs (Key Performance Indicators) of the implementation process [66], which allows for systematic standardization of methods and teaching materials used in training [67], and for adapting the information transfer process to the individual predispositions of a particular individual (e.g., predominance of visual- or speech-based form of information, the speed of information transfer, etc.) [68]. An additional noteworthy aspect in the field of production engineering is the possibility of providing a training stand able to take its parameters into account at the stage of simulating the influence of the new employee on the production line operation. This will be possible with the help of the Interaction Quality Sensor (IQS) proposed in this article, which introduces the implementation of Hybrid INTelligence (HINT) architecture for intelligent control systems, enclosing the prioritization of information channels and optimizing both the apprenticeship of unskilled workers and the performance of employees.
Industry 5.0 builds on the previous industrial revolutions but focuses on integrating new digital technologies to create an intelligent and flexible production system. Industry 5.0 aims to create an intelligent factory that is fully integrated and allows employees, machines, and IT systems to collaborate in a more harmonious way. This integration can lead to improved efficiency, reduced waste, and increased sustainability. The presented HINT system follows this trend by implementing the proposed adaptive mechanism in the area of information channels used in HMI. The HINT system uses multimedia information (e.g., speech, images, videos), as well as optional co-bot support, especially for applications with foreign and/or unqualified employees.

2. Materials and Methods

Production quality, often depicted by the technical quality of a product, seems to be quite an intuitive term. However, the evaluation of employee training quality is not trivial [69], and it may be not suitable for intelligent control of the training process unless it is properly approached, defined, and measured. This ‘training quality’ can be perceived as a result of (1) the training/teaching process (initial basic training as well as the mastering performed later) and (2) the personal predispositions of a particular employee, play an essential role in the decision-making process of the HINT system algorithm relaying especially on sensing capabilities of the IQS. The production line‘s sensors provide the knowledge of the apprentice’s work quality [64] and enable adjusting the steering parameters for HMI guidance delivery channels, conceivably due to the HINT system’s capability of full integration into the factory processing event bus. Based on the available information, the HINT system is designed to be able to distinguish alternate information channel scenarios and to utilize a control algorithm optimized with regard to a particular apprentice and the situation. The IQS delivers the necessary data by comparing patterns of behavior saved in the set of basic instructions against the behavior of the worker and expected quality in advancements of current activity. The optimization process of selection of the best-performing channel to the situation conditions was already briefly introduced in [64], and presented by comparing it to the concept of Hanoi Towers. Meanwhile, the study presented in this article refers to the real-world conditions of a local furniture factory and the conclusions drawn from real-world use-case implementation and integration with an existing quality control system, by enhancing the functionality of an existing apprentice workstation with HMI instruments, and upgrading it to a quality- and process-related research station. The factory was categorized as having: (a) medium complexity of operation producing wooden windows and doors for civil houses—manually assembled using prefabricated parts and specialized tools, (b) a significant level of job quitting individuals just after initial training, and (c) a long (unacceptable) time of training before reaching the required level of self-sufficient proficiency at work.
An important part of the experiment was the customized HINT system software, prepared and installed in the HMI workstation (see Section 2.4), providing work guidance/instructions to novices in textual, vocal, and visual form (both static and moving images)—in frames of a tutorial, especially considering preferences and the predispositions of the individuals, while recording their choices and performance (language, menu navigation, information channels choices, perception delay [70], assembly (execution) time, etc.). In an emergency (when no further steps of assembly could be reached), there was a possibility of a two-way voice connection to the foreperson for instructions/support. Communication between HMI and workers could be done by voice commands or touch screen. The transcripts of all interaction events were visible to the manager (foreperson) in a web browser window, along with the camera top view of the workstation, and the current state of all workstation screens. The foreperson was able to interfere with the communication of the system and the worker, by using a web chat window, to make adjustments (or even overtake) the system’s guiding role for a while. This functionality was also designed to introduce amendments that could actively retrain the underlying neural network model of the expert system (Figure 1). During the experiment run, it was recorded which elements of the system were used by the apprentice, for how long, and how many times (the most important aspect of the data collected was the use of information channels). The acquired performance data were linked to the result of the final assembly quality check, which allowed conclusions to be drawn. In pursuit of remarkable level of User eXperience of man–machine communication, the authors proposed a solution incorporating the following ideas [64]:
  • AI system, with a predefined fact database containing topics within an expert knowledge domain;
  • The ability to extend/supplement the system’s knowledge by adding new facts and relations;
  • A human-in-loop expert assistance for the approval of new knowledge and for emergency situations.

2.1. Hybrid Intelligence System (HINT) Architecture

The concept described in the current research aims to enable the adaptation of a system to users’ preferences and to improve the operation of the system as a result of the analysis of the process data. The system analyzes the initial stages of interaction and the user’s preferences (choice tendencies) and adjusts the presentation of content (resources) according to a personalized interaction model. The patterns are based on the data from previous interaction instances.
The developed HINT system allows for a variety of architecture configurations corresponding to the characteristics of the implementation requirements in a given production environment. As part of the system validation, the HINT system was implemented in the woodworking industry, specifically in a company producing wooden joinery. The specificity of the company focused on the implementation of short series and products, and even the production of personalized products, is a good training ground for the developed system.
The implementation of the HINT system, which is especially important in the face of the new multilingual nature of the labor market, takes the full benefits of the modular system architecture (presented as a simplified diagram in Figure 1). The following modules are involved in the personalization of user interaction: STT, TTS, data repositories, quality assurance module, and others.
The HINT system has been designed to be modular, allowing it to be adapted to customer requirements. One of the modules is responsible for system integration with external devices and systems. As an example of its versatility and modularity, the ABB YuMi co-bot can be used, which was successfully applied in data acquisition for active verification of assembly quality (as presented in Section 2.4).

2.2. Dual Sensor Component: Interaction Quality Sensor Merged with User Experience Sensor

The presented implementation uses the algorithm described in [64], designed as a dual mode sensor including User Experience (UX) sensor and Interaction Quality Sensor (IQS). The idea and theoretical basis of this sensor are described as an analogy to the towers of Hanoi (see Figure 2).
This is initiated automatically without the knowledge and awareness of the system user—which is fulfilling the role of the User Experience Sensor.
The primary objective is to reconsider the adaptation of the information channels to best-performing in the context of individual preferences of the user on the basis of the readouts of IQS/UX sensors. At a glance, the best-performing channel is selected after analysis of the retrospective data of previous stages of completed work as presented in Figure 2 using the Hanoi Towers approach. Exemplary adaptation of HMI guidance delivery channels in case of lack of progress (no switching to the next stage) in a given assignment would proceed as follows (described in detail later in Section 2.4):
  • If a trainee is stuck in a given stage and decides to try a different information channel, which is visualized in the Hanoi Tower as the same width of disks (the tower does not narrow up, the subsequent disk is of the same width, which represents a repetition of the stage);
  • The work performance at a given stage performed by a given trainee is acquired by the proposed IQS and these data increase the knowledge base about the trainees’ performance per stage;
  • The best-performing channel is considered for pre-selection as the default one for future users.
The secondary objective is to detect probable faults in users’ work as quickly as possible by comparing the path and timing of activated sections on the video stream between recorded traces of workers and saved patterns within the instruction. The found discrepancy is marked for verification and inspection in case of any automated quality checking system within the production line. Later, aggregated data of discrepancy of a larger count of complete users’ interactions are analyzed in terms of the quality of the interaction. High values would mean that instruction is not explicit or does not fit the personal capabilities of an individual.
In addition, data from this process can be used to optimize the process according to selected criteria, e.g., time, process quality, or both of these factors at the same time. Testing the performance of this sensor in optimization issues will be the subject of future research.
The process of building the tower of Hanoi can be used for representing any process implemented in the system. For proper analysis of interaction processes depicted using Hanoi towers (as in Figure 2), the following assumptions should be made:
  • The diameters of the rings placed on the tower are adjusted to the number of stages in the analyzed process;
  • The largest diameter ring corresponds to the first stage of the process, with rings gradually decreasing in size with each subsequent stage;
  • The height of each ring represents the duration of execution of each stage of the process;
  • The colors of the rings represent the particular chosen information channel, such as text, images, videos, or consultation with an expert.

2.3. Vision System as an Additional Detector of Process Anomalies

The IQS proposed in this article can optionally be equipped with a visual analysis subsystem of the correctness of assembly process stages, in order to facilitate, streamline, and automate the verification of the technical correctness of a given process stage. The inclusion of a video supervision subsystem significantly supports the operation of IQS, offering additional insight into the quality of work (or employee training).

2.4. Adaptive Information Channel Customization Module Based on the “Hanoi Towers” Concept

Quality control may be defined as a process that helps a company make sure that it creates quality products and that its employees make minimal mistakes. Therefore, in order to achieve this goal, two mechanisms have been distinguished: Passive Quality Control (Figure 3) (mechanisms supporting the employee by providing them with the necessary information, forcing the correct order of operations, etc.) and Active Quality Control (understood as active control and verification). The proposed implementation uses a vision system that analyzes the correct assembly of the key node.
During the research and development of the HINT (Hybrid INTeligence) system, (created as part of research at Weegree company, and implemented, tested, and validated at Halupczok company), numerous solutions were designed and applied in order to support the new employees during the process of getting acquainted with the new workplace and to support the quality control during the training and the performance of work. The HINT system is capable of using a co-bot to support active quality control (Figure 3B) by using vision processing. Passive quality control (Figure 3A) is carried out by enforcing the correct order of activities and is supervised by the HINT system.
The quality assurance module uses two mechanisms implemented in the HINT system: (1) passive quality control, which is carried out by enforcing the correct order of activities (stages) (supervised by the HINT system), and (2) active quality control (presented in Figure 3) using video analysis. In one of the test implementations of the system in the woodworking industry, a vision sensor integrated into a SmartGripper attached to the ABB YuMi co-bot (Figure 4) was used. It acquires visual data for image processing enabling the assessment of the correctness of the key stages of assembly of the wooden window sashes and frame. The YuMi co-bot operates in accordance with the guidelines of the ISO 10218-1 standard [71] for safety of people in a previously defined (at the stage of designing the workplace) workspace [72].
The HINT system, in the case of an employee’s need to contact a supervisor (or other competent person regarding the performed tasks), provides the possibility of a remote teleconference connection with an expert. This mechanism supports and implements the above-mentioned active quality control. Employees were able to contact an expert (called Avatar) when they encountered a difficult problem. The assembly training was monitored independently. During the initial phase of testing, before optimization, employees were given the autonomy to decide when to utilize the Avatar assistance. However, after the optimization, the HINT system was able to identify the Avatar to be the most appropriate information source and prioritized it higher than other information channels.
The analyzed experimental interactive assisted window assembly process consisted of twenty separate assembly steps, and was carried out at the Halupczok company production facility. A number of inexperienced people (with no prior experience in window assembly) faced the challenge of completing the task with the help of the interactive stand (presented in Figure 5). They were briefly instructed on its HMI interface, available information channels, including the last option (which would on the one hand make the assembly process possibly take longer, but on the other hand could be helpful in avoiding mistakes or damage)—the Avatar support (Figure 6).

2.5. User Interaction Adaptive Model Implementation

The Hybrid Intelligence System (HINT) was first introduced theoretically in [64], where the authors presented a novel user experience optimization concept and method, named the User Experience Sensor. The concept presented there enabled the selection of the most effective channels of information transfer [73] available to be chosen from for any given stage:
  • Text-based information (description);
  • Speech (TTS synthesized speech audio);
  • Images (visual guide with photos or diagrams);
  • Videos (mostly available only for stages that are difficult to explain);
  • Video call to the senior staff or process supervisor (VOIP audio connection with the additional top-view camera of the workbench).
Choosing and switching between available information channels (forms of information for every given stage of window assembly) helps to optimize the efficiency of a production line (for a given employee) or of training of a particular new employee.
In the implementation of the HINT system, the concept of quick employee onboarding—regardless of their current technical competencies and regardless of what languages they speak—was an absolute priority. The specificity of the local labor market consists of the availability of workforce; however, with limited knowledge of the local language (in this case, Polish). The system has a module that allows you to quickly change the predefined languages (languages of text messages, verbal messages, visual content). Verbal communication with the system also requires access to the STT service in the selected language.
An important element of the employee support system (Figure 5), especially at the initial stage, is the ability to contact an expert directly using the interactive workbench (Figure 6), and while multilingualism is supported, it is possible to translate the user’s and/or Avatar’s statements on the fly in the event of a language barrier using STT and translation and TTS. A translation module based on the Google Cloud service is used for this.
At the beginning of every training (the system operation), the personal preferences of individual employees are unknown. This is a result of individual cognitive preferences and so-called Default Bias [74,75] in the user expectancies towards interactive systems or kiosks. Therefore, the system starts working on the default configuration of information channels used to present content (Figure 7A). As a result of the analysis of data from the first stages (e.g., typically the first three stages, but the range can be adjusted), the system compares the particular employee’s interaction model and, if it deviates from the proposed model, reconfigures the information channels to adapt to the employee’s individual preferences (Figure 7B).
This tactic can be used many times during the user’s activity after collecting and analyzing each portion of information about the preferred information channels used. This mechanism also modifies the statistical model of the default configuration of information channels, which can be modified for use with selected cohorts (e.g., age- or language-related groups). The “default configuration of information channels” is understood as a predefined information channels-related communication scheme consisting of available channels (audio, video, graphics, and selected language) prioritized in a predicted/presumed order. When the system determines that the window fittings have been properly installed, it selects as default those forms of providing information that reduced the work time of the trained person while correctly completing the assembly stage [64]. The system was prepared to operate in four predefined languages or in the case of calling an avatar that does not speak the language of the learner, we have a multi-step process consisting of: STT—Google Translator—Avatar answer—STT—Google Translator—TTS. The adaptive mechanism of the method of choosing appropriate information channels supports the minimization of the negative effects of the Default Bias, which consists of imposing on the user an inadequate (not optimal) information channel choice, resulting from the statistical analysis of the generalized behavior of cohorts (Figure 7).

3. Results

The HINT system, presented in Section 2 (and its architecture—in Section 2.1), was included in the design of the prototype of the Adaptive information channel customization module (Section 2.4), which enabled the possibility to perform experimental research on the User interaction adaptive model (Section 2.5). The results of the trials—using HINT to leverage the efficiency and quality of training of new (unqualified) employees for a particular task—are presented below.

3.1. Use Case 1—HINT as a Window Assembly Assistant

The first use case process—onboarding of untrained employees using the HINT-equipped windows assembly interactive assistant—was conducted twice: once before optimization and once after, and the employees were not trained beforehand. The progression of each stage was recorded. Additionally, each user was asked to fill out a process evaluation questionnaire, which served as the foundation for the User eXperience (UX) evaluation. After finishing each HINT-assisted window assembly training session, the quality of each window assembly was assessed.
The experiment participants were chosen at random from a group of volunteers with no prior experience in window assembly. They were only given the HINT system, available on a touch screen with textual instructions and images for each assembly stage, and videos available for the majority of steps. Each employee was instructed how to interact through voice commands and the touch screen.

3.2. Use Case 2—HINT with a Window Assembly Co-Bot

The proposed HINT system was also tested with the use of ABB YuMi co-bot (see Figure 4) to investigate its purposefulness in operation with co-bots (cooperative robots). The YuMi co-bot was intended to help the human employee in the window assembly learning process, by participating in the assembly. However, it turned out that the inclusion of a robotic companion adds some complexity to the learning process. The basic part of the research (analysis, comparison, and optimization of the information channels in the training process) was designed and conducted without a co-bot and contributed the most interesting and valuable insights into HINT-assisted employee training. The inclusion of a co-bot introduces new interactive training options, extending the set of available information channels.

3.3. Analysis of the Learning Process Using HINT

The impact of the learning process using the HINT system depends on the participants’ perception of the difficulties of the technological process as well as the quality of the documentation prepared by a foreman (supervisor of the teaching procedure and expert) for mastering that process. For the window assembly process, which was used as the research use case basis for the analysis of the HINT system operation, the trainees/participants stated that the performed activities were difficult or very difficult, and the quality of the prepared documentation was good or very good. This allows for the conclusion that the perception of the quality of the instructions translates into actual actions resulting in the quality of window installation. This subjective assessment was confirmed during the real-life experimental verification. The correlation coefficient between the perceived quality of the instruction and the verified quality of the assembly was 0.69—both for the assemblies performed without the HINT system (only with paper documentation: text, illustrative photos, or in-person dialogue with an expert), as well as in the case of its use within the training process (information channels: text, illustrative photos, voice messages, illustrative video, or remote video conference with an expert).
Important information is that in the case of working with the HINT system, the ability to adjust information channels resulted in a significant decrease in the need to use time-consuming contact with an expert. The average time of a single contact with an expert decreased from about 62 s if the HINT system was not used and to 26 s if this system was used (Figure 8).
In the case of using the HINT system, an interesting information source usage specificity on window assembly can be observed. This specificity is illustrated by the Figure 9, which in a synthetic way shows the use of the information channels by all participants/trainees as the window assembly process progresses. Such determination of the independent variable results from the fact that there are significant discrepancies in the interaction with the HINT system by various persons. The smallest number of interactions was 10, while the largest number was 92.
Figure 9 shows that interactions involving the use of the vocalized description of an assembly step (blue) occur uniformly throughout the assembly process. At the same time, as the window assembly process progresses, the use of videos (green) increases (the number of uses is numerically highest at the end of assembly). In the initial stages of assembly, pictures and diagrams constitute a vast majority of information channel choices, which decreases (at about 2nd/3rd of the assembly process), along with the “discovery” of other, in the opinion of the learners, more advantageous forms of transferring information. The figure also shows that the repetition of the information transfer process (cyan) usually occurs in the middle of the assembly process and is not used at the beginning when the assembly steps depicted seem to be easy for the assembler. This may also be caused by fatigue of using one information channel, mitigated by using another one; however, the authors tend to see this situation rather as a search for better explanation of assembly steps. Asking for help from the supervisor (magenta) starts to appear about halfway through the process and initially aligns with the repetition of information delivery and then supplants the repetition process as some trainees have found this assistance more useful than repetition of the information delivery process in any other form. Table 1 contains detailed information on the number of uses of various forms (channels) of information retrieval by 10 selected unqualified participants of the window assembly experiment involving the use of the HINT system.
Table 1 shows that there were people who often used different information channels, e.g., participants 1 and 4. There were also people who preferred one specific form (channel) of information, for example, participant 6 and 9.

4. Discussion

The purpose of this paper was to present an implementation of Hybrid INTelligence architecture for Intelligent control systems (HINT), enclosing the prioritization of information channels, and optimizing both the apprenticeship of unskilled workers and the performance of employees.
The HINT system presented in the article and its exemplary implementation in the production environment is an example of a solution that naturally fits into the industry 4.0 concept. The modularity of the system facilitates its adaptation to particular use-case-specific implementation requirements, and at the same time enables the system to be extended with new functionalities. An example is the adaptive content customization module based on the concept of “Towers of Hanoi” (presented in [64]). Integration of applications originating from diverse implementation areas into common quality-oriented training systems (similar to the HINT presented in this study) seems to be the transcendent direction in which many other employee-supporting systems will follow.
They contribute to an increase in employee efficiency, ensure the quality of performed tasks and enable the labor market to absorb new employees (despite their initial lower competencies and/or a language barrier).
Looking into the future of development of processes containing a human element and trying to visualize the latest human-centric technological trends, the proprietary HINT system presented in this study offers the opportunity to improve the quality of training, as well as the effectiveness of human work and enables easier adaptation of a person in the process of performing the required activities by selecting the most effective information channels and methods of knowledge transfer, carried out without the participation of experts.
The HINT system was developed with thought and intention to leverage the teaching effectiveness of multimedia information channels, such as speech, images, and videos (and optionally the assistance of a co-bot), particularly for applications with inexperienced employees (with limited skills or with language barriers). HINT aims to enable employees and aid enterprises in efficiently integrating new hires into the works and processes performed.
The proposed solution in the form of the HINT system is part of the future development of robotization, automation, and the upcoming Industry 5.0. The HINT system has many possibilities that enable the quick assimilation of information required in the implementation of tasks in places where employees are required for various reasons, including economic ones. Current studies of the use of the HINT system indicate an increase in the speed and efficiency of knowledge acquisition.
Industry 5.0 gives new opportunities to obtain high added value of human capital as an element that gives value to manufactured products [76]. The idea of Industry 5.0 emphasizes the need for cooperation between man and machine, while pointing to the importance of humans in the manufacturing processes. The idea of people and machines working together leads to flexible and effective business models [77,78,79,80,81,82,83].

4.1. HINT’s Placement in the Kaleidoscope of Robotics’ Future

Currently, enterprises operate in times of rapid technological progress, in particular the development of digitization driven by the need to reduce energy consumption, maximize efficiency, and react faster to changes in the immediate and global environment. The events of 2021–2022 significantly changed the list of issues to be considered when planning business continuity, i.e., dynamic adaptation to new conditions and ensuring sustainable development.
Ideas based on automation, robotization, and artificial intelligence are already used in the Industry 4.0 trend and will, thus, dominate the upcoming industry of the future. Currently, the possibilities of new technologies and the technological potential of robots have reached an unbelievable level of development, and their general availability in connection with large adoption (almost indispensable) in enterprises has a critical impact on the development of many industries. Robotics, automation, and AI are already changing the quality of life in a wide range of activities [84] and the way work is performed, increasing the level of efficiency and safety of activities and processes, improving the standard of services, customer service, and logistics, and improving the supply chain.
Cooperative teams, consisting of humans and machines (robots), can be seen to constitute interconnected cognitive entities, able to include human communication using natural language. With the communication performed purely using voice or text communication, as well as answering questions and giving recommendations for action, cooperative systems are able to use their knowledge and capabilities to contribute to increasing the quality of manufactured products, optimizing the implementation of activities and making better decisions (both strategic and tactical) by the management staff [85]. The progress of digitization, new technologies within the Industry 4.0 and the idea of Industry 5.0, automation, robotization, and use of AI require the use of the Internet, which results in the dawn of a new era of human–robot interaction, IoT, security, cybersecurity, work performance, functioning, and operation.
In the last few years, new types of interactions between humans and machine learning algorithms have emerged, which can be grouped under the umbrella term human-in-the-loop (HITL) [86,87]. HITL is a branch of artificial intelligence that requires the involvement of both humans and machines to create machine learning models [88]. HITL illustrates a process where a machine or computer system is unable to solve a problem on its own and requires human intervention. The human, then, becomes the element that trains, adapts, tests, and in many cases also controls the system’s algorithms. The HITL process is a continuous feedback loop, which means that every training-and-test task is fed back into the algorithm. HITL aims to achieve what neither man nor machine can achieve in isolation, i.e., by functioning independently. An effective HITL system is designed in such a way as to allow many people to contribute information to the learning process (at any time), and the person giving feedback is usually responsible for making the final decisions about the learning process [76,89,90]. By applying the HITL concept, HINT becomes more capable of performing a variety of tasks by facilitating employees’ cognitive (including knowledge) and physical (mainly manual and motor) skills, while profiting from the robots’ ability to perform repetitive tasks and tedious activities [91]. The advantage of using HINT is the ability to parameterize the quality of the job instructions provided, and more precisely to identify elements that need improvement because they are incomprehensible or complicated for the employee, which is visible through the excessive use of communication channels around one stage. If the obtained information is used to update work instructions, the instruction standardization process can be carried out. The ubiquitous profiling (it is common in the world of the Internet to learn the preferences of the Internet user for marketing purposes) is also reflected in HINT, but in a fruitful way for everyone—allowing for the personalization of the process of transferring knowledge to the user, enabling them to learn quickly, and hypothetically reducing their stress and tension. HINT is a system that connects a cooperating man with a machine into a team and provides the function of a supervisor, thus relieving the qualified management staff.

4.2. Safety of the System (Human and Robots Cooperation)

Measures to ensure safety during human–machine cooperation are described in the ISO 10218 standard [71,92] (entitled “Robots and robotic devices—Safety requirements for industrial robots”). This is an international standard for the safety of industrial robots, developed by ISO/TC 184/SC 2 “Robots and robotic devices” in parallel with the European Committee for Standardization. It consists of two parts: (1) “Robots”, describing the requirements for robots, and (2) “Robotic system and integration”. Together, these standards contain requirements for robotic systems, including systems cooperating with robots. The standard is aimed at robot manufacturers, integrators, and builders of workstations containing robots and co-bots. According to the [93], a collaborative robot workstation “should be considered as a machine or assembly of machines, depending on the degree of complexity”, which implies the necessity to extend the scope of considerations—from “a robot” to “the system”.

4.3. The Co-Bots’ Future and the Road Map to Industry 5.0

Presently, a crucial aspect in the advancement of businesses is the integration of machines, robotic systems, people, and processes [94,95,96], resulting in an increased utilization of co-bots. The incorporation of robots and co-bots in businesses brings numerous benefits for both the enterprises and their employees. In addition, the lack of qualified employees means that companies increasingly decide to automate processes, including the use of robots, which begin to play the role of collaborators, with their potential recognized by entrepreneurs. Human–robot collaboration sets a new direction of activities in which human–robot cooperation takes place [91]. Collaborative robots (co-bots) are machines with technical precision in performing tasks, designed to work safely with people in human-centric environments [97,98,99,100,101,102]. Co-bots, due to their flexibility, neatness, and light mechanical structure (low weight, small size, ease of use, and being extremely precise) can be introduced in any enterprise and used for various processes and tasks, practically at any time [103,104,105,106].

5. Conclusions

This article is the summary and the result of a new approach to the optimization of man–machine interaction in collaborative, Industry 5.0-compliant workplaces. The authors propose not only sensing the quality of individual tasks as “building blocks” of the process’ representation, but also adding a supervising layer able to evaluate, manage, and optimize the execution of individual blocks (stages) as well as the whole interaction.
One of the results of the research is the ability to model the window assembly process (or any given work task) using the proposed quality parameters, visualized as the “building blocks” of a specific height and width.
The most important research result (and the research contribution) is the ability to use the proposed process model to automate the optimization, taking into consideration the individual preferences of particular participants, easily introducing human-centricity and collaborative intelligence into the Industry 5.0 workflow.

6. Future Work

The main challenge for upcoming research will be the adaptation and profiling of the proposed architecture to specific use cases of new workplaces. Further improvements and advancements will include, inter alia: centralized AI engine for numerous domain-restricted agents [107], distributed implementation for the centralized AI engine [108], load optimization for the distributed engine [109,110,111,112], integration of interactive artificial agents (including co-bots), and possibly a better data representation model created using information retrieval logic and a specified description language [113,114] instead of a static database of facts and relations. An alternative approach would be to follow the mainstream of current AI research and to implement a GPT-3-based chatbot [115]; however, this approach either needs reasonable funding or will generate a general model not suitable for domain-specific professional uses. Therefore, the future of assisted quality-oriented employee training systems remains an open topic.

Author Contributions

Conceptualization, A.G., R.B., B.K. and M.P.; methodology, A.G., R.B., B.K. and M.P.; software, B.K.; validation, A.G., R.B. and B.K.; formal analysis, R.B.; investigation, R.B.; resources, B.K.; data curation, A.G., B.K. and M.P.; writing—original draft preparation, M.P., A.G., J.R., B.K. and R.B.; writing—review and editing, M.P., R.B., J.R., A.G. and B.K.; visualization, A.G. and B.K.; supervision, A.G. and R.B.; project administration, R.B.; funding acquisition, A.G. All authors have read and agreed to the published version of the manuscript. All authors contributed equally to this work.

Funding

Project Title: Industrial research and development works in Weegree Sp. z o.o. (LLC) Sp.K. (LLP) on the system of hybrid intelligence supporting smart specializations of the Opole region. Agreement number: RPOP.01.01.00-16-0072/16-00. Project co-financed by the European Union from the European Regional Development Fund within the framework of the Regional Operational Programme of the Opolskie region for the years 2014–2020: Priority Axis 01 Innovation in the Economy, Measure 1.1 Innovation in Enterprises.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Datasets and source code are unavailable due to intellectual property.

Conflicts of Interest

The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Paschek, D.; Mocan, A.; Draghici, A. Industry 5.0–The expected impact of next industrial revolution. In Proceedings of the Thriving on Future Education, Industry, Business, and Society, Proceedings of the MakeLearn and TIIM International Conference, Piran, Slovenia, 15–17 May 2019; pp. 15–17. [Google Scholar]
  2. Maddikunta, P.K.R.; Pham, Q.V.; Prabadevi, B.; Deepa, N.; Dev, K.; Gadekallu, T.R.; Ruby, R.; Liyanage, M. Industry 5.0: A survey on enabling technologies and potential applications. J. Ind. Inf. Integr. 2022, 26, 100257. [Google Scholar] [CrossRef]
  3. Wu, W.Y.; Chiag, C.Y.; Wu, Y.J.; Tu, H.J. The influencing factors of commitment and business integration on supply chain management. Ind. Manag. Data Syst. 2004, 104, 322–333. [Google Scholar] [CrossRef]
  4. Rut, J.; Kulińska, E. Optimization of the material and information streams flow in the manufacturing company. Mark. Rynek 2017, 7, 616–624. [Google Scholar]
  5. Gurcan, F.; Cagiltay, N.E.; Cagiltay, K. Mapping human–computer interaction research themes and trends from its existence to today: A topic modeling–based review of past 60 years. Int. J. Hum.-Interact. 2021, 37, 267–280. [Google Scholar] [CrossRef]
  6. Grudin, J. Anticipating the future of HCI by understanding its past and present. In Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow Scotland, UK, 4–9 May 2019; pp. 1–4. [Google Scholar]
  7. Khoma, V.; Khoma, Y.; Brydinskyi, V.; Konovalov, A. Development of Supervised Speaker Diarization System Based on the PyAnnote Audio Processing Library. Sensors 2023, 23, 2082. [Google Scholar] [CrossRef]
  8. Danilevsky, M.; Qian, K.; Aharonov, R.; Katsis, Y.; Kawas, B.; Sen, P. A survey of the state of explainable AI for natural language processing. arXiv 2020, arXiv:2010.00711. [Google Scholar]
  9. Zhang, P.; Li, N.; Scialdone, M.J.; Carey, J.M. The Intellectual Advancement of Human–Computer Interaction Research: A Critical Assessment of the MIS Literature (1990–2008). AIS Trans. Hum.-Comput. Interact. 2009, 1, 55–107. [Google Scholar] [CrossRef] [Green Version]
  10. Yun, Y.; Ma, D.; Yang, M. Human–computer interaction–based decision support system with applications in data mining. Future Gener. Comput. Syst. 2021, 114, 285–289. [Google Scholar] [CrossRef]
  11. Arrieta, A.B.; Díaz-Rodríguez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; García, S.; Gil-López, S.; Molina, D.; Benjamins, R.; et al. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 2020, 58, 82–115. [Google Scholar] [CrossRef] [Green Version]
  12. Neville, S.J. Eavesmining: A critical audit of the Amazon Echo and Alexa conditions of use. Surveill. Soc. 2020, 18, 343–356. [Google Scholar] [CrossRef]
  13. Gao, Y.; Pan, Z.; Wang, H.; Chen, G. Alexa, my love: Analyzing reviews of amazon echo. In Proceedings of the 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), Guangzhou, China, 8–12 October 2018; pp. 372–380. [Google Scholar]
  14. Major, D.; Huang, D.Y.; Chetty, M.; Feamster, N. Alexa, who am I speaking to?: Understanding users’ ability to identify third–party apps on amazon Alexa. ACM Trans. Internet Technol. (TOIT) 2021, 22, 1–22. [Google Scholar] [CrossRef]
  15. Larsonneur, C. Smart and/or diverse: The paradoxes of machine processed language. Hybrid. Revue Arts et Médiations Hum. 2021. [Google Scholar] [CrossRef]
  16. Lardinois, F. Google Launches an Improved Speech–to–Text Service for Developers. Available online: https://techcrunch.com/2018/04/09/google-launches-an-improved-speech-to-text-service-for-developers (accessed on 26 November 2022).
  17. Ożarowska, I. Google Cloud Speech–to–Text—What Is It, and How to Use It. Available online: https://fotc.com/blog/speech-to-text-what-is (accessed on 26 November 2022).
  18. Dale, R. The return of the chatbots. Nat. Lang. Eng. 2016, 22, 811–817. [Google Scholar] [CrossRef] [Green Version]
  19. Hoy, M.B. Alexa, Siri, Cortana, and more: An introduction to voice assistants. Med Ref. Serv. Q. 2018, 37, 81–88. [Google Scholar] [CrossRef]
  20. Li, B.; Sainath, T.N.; Narayanan, A.; Caroselli, J.; Bacchiani, M.; Misra, A.; Shafran, I.; Sak, H.; Pundak, G.; Chin, K.K.; et al. Acoustic Modeling for Google Home. In Proceedings of the Interspeech, Stockholm, Sweden, 20–24 August 2017; pp. 399–403. [Google Scholar]
  21. Gupta, M.P. Google assistant controlled home automation. Int. Res. J. Eng. Technol. 2018, 5, 2074–2077. [Google Scholar]
  22. Peng, C.Y.; Chen, R.C. Voice recognition by Google Home and Raspberry Pi for smart socket control. In Proceedings of the 2018 Tenth International Conference on Advanced Computational Intelligence (ICACI), Xiamen, China, 29–31 March 2018; pp. 324–329. [Google Scholar]
  23. Garn, B.; Schreiber, D.P.; Simos, D.E.; Kuhn, R.; Voas, J.; Kacker, R. Combinatorial methods for testing internet of things smart home systems. Softw. Test. Verif. Reliab. 2022, 32, e1805. [Google Scholar] [CrossRef]
  24. Zhang, N.; Mi, X.; Feng, X.; Wang, X.; Tian, Y.; Qian, F. Understanding and mitigating the security risks of voice–controlled third–party skills on amazon alexa and google home. arXiv 2018, arXiv:1805.01525. [Google Scholar]
  25. Cohn, M.; Segedin, B.F.; Zellou, G. Acoustic–phonetic properties of Siri–and human–directed speech. J. Phon. 2022, 90, 101123. [Google Scholar] [CrossRef]
  26. Kepuska, V.; Bohouta, G. Next–generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In Proceedings of the 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, 8–10 January 2018; pp. 99–103. [Google Scholar]
  27. Kim, D.; Kim, S.H.; Kim, T.; Kang, B.B.; Lee, M.; Park, W.; Ku, S.; Kim, D.; Kwon, J.; Lee, H.; et al. Review of machine learning methods in soft robotics. PLoS ONE 2021, 16, e0246102. [Google Scholar] [CrossRef] [PubMed]
  28. Azeem, M.; Haleem, A.; Javaid, M. Symbiotic relationship between machine learning and Industry 4.0: A review. J. Ind. Integr. Manag. 2022, 7, 401–433. [Google Scholar] [CrossRef]
  29. Lins, R.G.; Givigi, S.N. Cooperative robotics and machine learning for smart manufacturing: Platform design and trends within the context of industrial internet of things. IEEE Access 2021, 9, 95444–95455. [Google Scholar] [CrossRef]
  30. Zhang, J.; Tao, D. Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things. arXiv 2020, arXiv:2011.08612. [Google Scholar] [CrossRef]
  31. Jadeja, M.; Varia, N. Perspectives for evaluating conversational AI. arXiv 2017, arXiv:1709.04734. [Google Scholar]
  32. Cravo, A.C.; Pereira, A.; Pereira, E.d.S. The challenges for knowledge workers in organizations to carry out sustainable organizational development. Rev. E-Tecnol. Para Compet. Ind. 2022, 15, 1–15. [Google Scholar] [CrossRef]
  33. Zhu, Y.; Wang, D.; Liao, G.; Liu, L.; Chen, Y.; Wang, L.; Yang, H.; Zhan, W.; Tian, Y. Description Strategy Selection in Collaborative Spatial Tasks. In Proceedings of the Man–Machine–Environment System Engineering, Beijing, China, 21–23 October 2022; pp. 427–433. [Google Scholar]
  34. Russell, N.; van der Aalst, W.M.P.; ter Hofstede, A.H.M.; Edmond, D. Workflow Resource Patterns: Identification, Representation and Tool Support. In Proceedings of the Advanced Information Systems Engineering, Porto, Portugal, 13–17 June 2005; pp. 216–232. [Google Scholar]
  35. Haase, T.; Keller, A.; Warschewske, F.; Woitag, M.; Sauer, S.; Berndt, D. Digital Assembly Assistance Systems: Methods, Technologies and Implementation Strategies. In Human–Technology Interaction: Shaping the Future of Industrial User Interfaces; Springer International Publishing: Cham, Switzerland, 2023; pp. 15–48. [Google Scholar] [CrossRef]
  36. Ing, L.Y.; Grossman, G.M. Robots and AI: A New Economic Era; Taylor & Francis: Abingdon, UK, 2022. [Google Scholar]
  37. Krstic, B. Identity, Image and Reputation as Determinants of Enterprise Competitiveness. Ekon. J. Econ. Theory Pract. Soc. Issues 2013, 59, 26–35. [Google Scholar] [CrossRef]
  38. Brito, T.; Queiroz, J.; Piardi, L.; Fernandes, L.A.; Lima, J.; Leitão, P. A machine learning approach for collaborative robot smart manufacturing inspection for quality control systems. Procedia Manuf. 2020, 51, 11–18. [Google Scholar] [CrossRef]
  39. Chen, J.; Mohammed, A.; Alexopoulos, T.; Setchi, R. Collaborative Human–Robot Assembly: Methodology, Simulation and Industrial Validation. In Proceedings of the Sustainable Design and Manufacturing, Split, Croatia, 14–16 September 2022; pp. 181–190. [Google Scholar]
  40. Gan, J.; Liu, L.; Qiao, G.; Zhang, Q. The role of robot adoption in green innovation: Evidence from China. Econ. Model. 2023, 119, 106128. [Google Scholar] [CrossRef]
  41. Awaworyi Churchill, S.; Inekwe, J.; Ivanovski, K.; Smyth, R. Human capital and energy consumption: Six centuries of evidence from the United Kingdom. Energy Econ. 2023, 117, 106465. [Google Scholar] [CrossRef]
  42. Buchanan, J.; Kronk, H. The Slow Adjustment in Tech Labor: Why Do High–Paying Tech Jobs Go Unfilled? Cent. Growth Oppor. 2023. [Google Scholar] [CrossRef]
  43. Skare, M.; de las Mercedes de Obesso, M.; Ribeiro-Navarrete, S. Digital transformation and European small and medium enterprises (SMEs): A comparative study using digital economy and society index data. Int. J. Inf. Manag. 2023, 68, 102594. [Google Scholar] [CrossRef]
  44. Payan-Carreira, R. Developing Critical Thinking in Higher Education: Is There a Reason to Change? In Proceedings of the Technology and Innovation in Learning, Teaching and Education: Third International Conference, TECH–EDU 2022, Lisbon, Portugal, 31 August–2 September 2022; p. 329. [Google Scholar]
  45. Perez-Encinas, A.; Berbegal-Mirabent, J. Who gets a job sooner? Results from a national survey of master’s graduates. Stud. High. Educ. 2023, 48, 174–188. [Google Scholar] [CrossRef]
  46. McLaren, J. Globalization and labor market dynamics. Annu. Rev. Econ. 2017, 9, 177–200. [Google Scholar] [CrossRef]
  47. Rutkowski, J. Rapid Labor Reallocation with a Stagnant Unemployment Pool. The Puzzle of the Labor Market in Lithuania; The Puzzle of the Labor Market in Lithuania, World Bank Policy Research Working Paper; World Bank: Washington, DC, USA, 2003. [Google Scholar]
  48. Ruiz, I.; Vargas-Silva, C. The labor market impacts of forced migration. Am. Econ. Rev. 2015, 105, 581–586. [Google Scholar] [CrossRef] [Green Version]
  49. Dominese, G.; Yakubovskiy, S.; Tsevukh, J.; Rodionova, T. Impact of international migration flows on the European union and Ukraine. Transit. Stud. Rev. 2020, 27, 83–98. [Google Scholar]
  50. Habarta, A. The Migration Crisis in Modern Europe. 2022. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4136829 (accessed on 26 November 2022).
  51. Shimada, A. Foreign worker participation in labor markets and the economy’s welfare. J. Policy Model. 2005, 27, 355–362. [Google Scholar] [CrossRef]
  52. Gupta, B.B.; Gaurav, A.; Panigrahi, P.K.; Arya, V. Analysis of artificial intelligence–based technologies and approaches on sustainable entrepreneurship. Technol. Forecast. Soc. Chang. 2023, 186, 122152. [Google Scholar] [CrossRef]
  53. Webster, C.; Ivanov, S. Robotics, artificial intelligence, and the evolving nature of work. In Digital Transformation in Business and Society; Springer: Berlin/Heidelberg, Germany, 2020; pp. 127–143. [Google Scholar]
  54. Vrontis, D.; Christofi, M.; Pereira, V.; Tarba, S.; Makrides, A.; Trichina, E. Artificial intelligence, robotics, advanced technologies and human resource management: A systematic review. Int. J. Hum. Resour. Manag. 2022, 33, 1237–1266. [Google Scholar] [CrossRef]
  55. Yawalkar, M.V.V. A Study of Artificial Intelligence and its role in Human Resource Management. Int. J. Res. Anal. Rev. (IJRAR) 2019, 6, 20–24. [Google Scholar]
  56. Rathi, R. Artificial intelligence and the future of hr practices. Int. J. Appl. Res. 2018, 4, 113–116. [Google Scholar]
  57. Barboza, C. Artificial Intelligence and HR: The New Wave of Technology. J. Adv. Soc. Sci. Humanit. 2019, 5, 715–720. [Google Scholar]
  58. Armstrong, M.; Taylor, S. Armstrong’s Handbook of Human Resource Management Practice: A Guide to the Theory and Practice of People Management, 15th ed.; Kogan Page Ltd.: London, UK, 2020; ISBN 978-074-949-827-6. [Google Scholar]
  59. Nie, P.y.; Yang, Y.C. Innovation and competition with human capital input. Manag. Decis. Econ. 2023, 44, 1779–1785. [Google Scholar] [CrossRef]
  60. De Felice, S.; Hamilton, A.F.d.C.; Ponari, M.; Vigliocco, G. Learning from others is good, with others is better: The role of social interaction in human acquisition of new knowledge. Philos. Trans. R. Soc. B Biol. Sci. 2023, 378, 20210357. [Google Scholar] [CrossRef]
  61. Giskeødegård, M.; Kjersem, K.; Jahn, N.; Rost, R. Tool or hassle?—Production workers evaluation of the potential of digital assistance systems on the shopfloor in shipbuilding projects. Cogent Eng. 2023, 10, 2161763. [Google Scholar] [CrossRef]
  62. Azam, M.S. How can we build human resources (HR) that deliver value? A systematic literature review of traditional and transformational HR roles. Glob. Bus. Organ. Excell. 2023, 42, 81–92. [Google Scholar] [CrossRef]
  63. Bøgh, S.; Hain, D.S.; Hansen, E.B.; Jensen, S.B.; Tvedebrink, T.; Jurowetzki, R. Predictive Analytics Applications for Small and Medium-Sized Enterprises (SMEs)—A Mini Survey and Real–World Use Cases. In The Future of Smart Production for SMEs: A Methodological and Practical Approach Towards Digitalization in SMEs; Springer International Publishing: Cham, Switzerland, 2023; pp. 263–279. [Google Scholar] [CrossRef]
  64. Gardecki, A.; Podpora, M.; Beniak, R.; Klin, B.; Pochwała, S. User Experience Sensor for Man–Machine Interaction Modeled as an Analogy to the Tower of Hanoi. Sensors 2020, 20, 4074. [Google Scholar] [CrossRef] [PubMed]
  65. Wu, S.; Wang, Z.; Shen, B.; Wang, J.H.; Dongdong, L. Human–computer interaction based on machine vision of a smart assembly workbench. Assem. Autom. 2020, 40, 475–482. [Google Scholar] [CrossRef]
  66. Caiazzo, C.; Nestić, S.; Savković, M. A Systematic Classification of Key Performance Indicators in Human–Robot Collaboration. In Proceedings of the Sustainable Business Management and Digital Transformation: Challenges and Opportunities in the Post–COVID Era, Belgrade, Serbia, 11–14 June 2022; pp. 479–489. [Google Scholar]
  67. Stenner, A.J.; Bland, J.D.; Hunter, E.L.; Cooper, M.L. The Standardized Growth Expectation: Implications for Education Evaluation. In Explanatory Models, Unit Standards, and Personalized Learning in Educational Measurement: Selected Papers by A. Jackson Stenner; Springer Nature: Singapore, 2023; pp. 1–15. [Google Scholar] [CrossRef]
  68. Herholz, S.C. Individual Predisposition for Learning and Neuroplasticity. J. Neurosci. 2013, 33, 15321–15323. [Google Scholar] [CrossRef] [Green Version]
  69. Komaki, J.; Heinzmann, A.T.; Lawson, L. Effect of training and feedback: Component analysis of a behavioral safety program. J. Appl. Psychol. 1980, 65, 261. [Google Scholar] [CrossRef] [PubMed]
  70. Chakraborty, G.; Kikuchi, D.; Sawamoto, J.; Yokoha, H. Perception delay and its estimation analyzing EEG signal. In Proceedings of the 2013 IEEE International Conference on Cybernetics (CYBCO), Lausanne, Switzerland, 13–15 June 2013; pp. 57–62. [Google Scholar] [CrossRef]
  71. ISO 10218-1:2011; Robots and Robotic Devices—Safety Requirements for Industrial Robots—Part 1: Robots. ISO: Geneva, Switzerland, 2011.
  72. Ruijten, P.A.; Cuijpers, R.H. Do not let the robot get too close: Investigating the shape and size of shared interaction space for two people in a conversation. Information 2020, 11, 147. [Google Scholar] [CrossRef] [Green Version]
  73. Ren, F.; Bao, Y. A review on human–computer interaction and intelligent robots. Int. J. Inf. Technol. Decis. Mak. 2020, 19, 5–47. [Google Scholar] [CrossRef] [Green Version]
  74. Momsen, K.; Schneider, S.O. Motivated Reasoning, Information Avoidance, and Default Bias; Working Papers in Economics and Statistics; University of Innsbruck: Innsbruck, Austria, 2022; pp. 1–43. ISSN 1993-4378. [Google Scholar]
  75. Soll, J.B.; Milkman, K.L.; Payne, J.W. A user’s guide to debiasing. Wiley Blackwell Handb. Judgm. Decis. Mak. 2015, 2, 924–951. [Google Scholar]
  76. Wang, B.; Tao, F.; Fang, X.; Liu, C.; Liu, Y.; Freiheit, T. Smart manufacturing and intelligent manufacturing: A comparative review. Engineering 2021, 7, 738–757. [Google Scholar] [CrossRef]
  77. Lu, Y.; Zheng, H.; Chand, S.; Xia, W.; Liu, Z.; Xu, X.; Wang, L.; Qin, Z.; Bao, J. Outlook on human–centric manufacturing towards Industry 5.0. J. Manuf. Syst. 2022, 62, 612–627. [Google Scholar] [CrossRef]
  78. Nahavandi, S. Industry 5.0–A human–centric solution. Sustainability 2019, 11, 4371. [Google Scholar] [CrossRef] [Green Version]
  79. Miraz, M.H.; Hasan, M.T.; Sumi, F.R.; Sarkar, S.; Hossain, M.A. Industry 5.0: The Integration of Modern Technologies. In Machine Vision for Industry 4.0; CRC Press: Boca Raton, FL, USA, 2022; pp. 285–300. [Google Scholar]
  80. Zizic, M.C.; Mladineo, M.; Gjeldum, N.; Celent, L. From industry 4.0 towards industry 5.0: A review and analysis of paradigm shift for the people, organization and technology. Energies 2022, 15, 5221. [Google Scholar] [CrossRef]
  81. Ćoćkalo, D.; Bakator, M.; Đorđević, D.; Vorkapić, M.; Stanisavljev, S. Industry 5.0: A new paradigm in manufacturing. In Proceedings of the 3rd Virtual International Conference Path to a Knowledge Society-Managing Risks and Innovation 2021, Virtual, 15–16 November 2021; Stanković, M., Nikolić, V., Eds.; ComplexSystem Research Center: Belgrade, Serbia, 2021; pp. 245–250. [Google Scholar]
  82. Xu, X.; Lu, Y.; Vogel-Heuser, B.; Wang, L. Industry 4.0 and Industry 5.0–Inception, conception and perception. J. Manuf. Syst. 2021, 61, 530–535. [Google Scholar] [CrossRef]
  83. Demir, K.A.; Döven, G.; Sezen, B. Industry 5.0 and human–robot co–working. Procedia Comput. Sci. 2019, 158, 688–695. [Google Scholar] [CrossRef]
  84. Qian, C.; Zhu, C.; Huang, D.H.; Zhang, S. Examining the influence mechanism of artificial intelligence development on labor income share through numerical simulations. Technol. Forecast. Soc. Chang. 2023, 188, 122315. [Google Scholar] [CrossRef]
  85. Lillywhite, B.; Wolbring, G. Auditing the impact of artificial intelligence on the ability to have a good life: Using well-being measures as a tool to investigate the views of undergraduate STEM students. AI Soc. 2023, 1, 1–16. [Google Scholar]
  86. Karwowski, W. International Encyclopedia of Ergonomics and Human Factors—3 Volume Set; CRC Press: Boca Raton, FL, USA, 2006. [Google Scholar]
  87. Munro, R. The rapid growth of Human–in–the–Loop Machine Learning. In Proceedings of the Workshop on Interactive Adaptive Learning, The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Ghent, Belgium, 14 September 2020. [Google Scholar]
  88. Mituniewicz, N. Human in the Loop, or This Is How Humans Help Artificial Intelligence… Get Smart. Available online: https://homodigital.pl/human-in-the-loop-hitl-automatyzacja-homodigital (accessed on 26 November 2022). (In Polish).
  89. Mosqueira-Rey, E.; Hernández-Pereira, E.; Alonso-Ríos, D.; Bobes-Bascarán, J.; Fernández-Leal, Á. Human–in–the–loop machine learning: A state of the art. Artif. Intell. Rev. 2022, 56, 3005–3054. [Google Scholar] [CrossRef]
  90. Netzer, E.; Geva, A.B. Human–in–the–loop active learning via brain computer interface. Ann. Math. Artif. Intell. 2020, 88, 1191–1205. [Google Scholar] [CrossRef]
  91. Turner, C.J.; Ma, R.; Chen, J.; Oyekan, J. Human in the Loop: Industry 4.0 technologies and scenarios for worker mediation of automated manufacturing. IEEE Access 2021, 9, 103950–103966. [Google Scholar] [CrossRef]
  92. ISO 10218-2:2011; Robots and Robotic Devices—Safety Requirements for Industrial Robots—Part 2: Robot Systems and Integration. ISO: Geneva, Switzerland, 2011.
  93. Safety of Systems with Collaborating Robots Part 1. Available online: https://fugai.pl/bezpieczenstwo-systemow-z-robotami-wspolpracujacymi-cz-1 (accessed on 26 November 2022). (In Polish).
  94. Hentout, A.; Aouache, M.; Maoudj, A.; Akli, I. Human–robot interaction in industrial collaborative robotics: A literature review of the decade 2008–2017. Adv. Robot. 2019, 33, 764–799. [Google Scholar] [CrossRef]
  95. Galin, R.R.; Meshcheryakov, R.V. Human–robot interaction efficiency and human–robot collaboration. In Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms; Springer: Berlin/Heidelberg, Germany, 2020; pp. 55–63. [Google Scholar]
  96. de Assis Dornelles, J.; Ayala, N.F.; Frank, A.G. Smart Working in Industry 4.0: How digital technologies enhance manufacturing workers’ activities. Comput. Ind. Eng. 2022, 163, 107804. [Google Scholar] [CrossRef]
  97. Wang, L.; Liu, S.; Liu, H.; Wang, X.V. Overview of human–robot collaboration in manufacturing. In Proceedings of the 5th International Conference on the Industry 4.0 Model for Advanced Manufacturing, Belgrade, Serbia, 1–4 June 2020; pp. 15–58. [Google Scholar]
  98. Evjemo, L.D.; Gjerstad, T.; Grøtli, E.I.; Sziebig, G. Trends in smart manufacturing: Role of humans and industrial robots in smart factories. Curr. Robot. Rep. 2020, 1, 35–41. [Google Scholar] [CrossRef] [Green Version]
  99. Arents, J.; Abolins, V.; Judvaitis, J.; Vismanis, O.; Oraby, A.; Ozols, K. Human–robot collaboration trends and safety aspects: A systematic review. J. Sens. Actuator Netw. 2021, 10, 48. [Google Scholar] [CrossRef]
  100. Nadeau, P.; Giamou, M.; Kelly, J. Fast Object Inertial Parameter Identification for Collaborative Robots. arXiv 2022, arXiv:2203.00830. [Google Scholar]
  101. Kralj, D.; Aralica, K. Industry 5.0 from the Perspective of Safety at Work. In Occupational Safety and Health, Proceedings of the 8th International Professional and Scientific Conference, Zadar, Croatia, 21–24 September 2022; Karlovac University of Applied Sciences: Karlovac, Croatia, 2022; pp. 114–120. [Google Scholar]
  102. Gualtieri, L.; Rauch, E.; Vidoni, R. Development and validation of guidelines for safety in human–robot collaborative assembly systems. Comput. Ind. Eng. 2022, 163, 107801. [Google Scholar] [CrossRef]
  103. Álvarez, D.D.; Væhrens, L. Collaborative robots for smart production in SMEs. In The Future of Smart Production for SMEs; Springer: Berlin/Heidelberg, Germany, 2023; pp. 363–369. [Google Scholar]
  104. Paluch, S.; Wirtz, J.; Kunz, W.H. The Service Robot Revolution; Edward Elgar Publishing: Camberley, UK, 2022; Chapter 20; pp. 296–308. ISBN 978-180-037-564-2. [Google Scholar]
  105. Holm, M.; Schnell, M. Challenges for Manufacturing SMEs in the Introduction of Collaborative Robots. In Proceedings of the 10th Swedish Production Symposium (SPS2022), Skövde, Sweden, 26–29 April 2022; pp. 173–183. [Google Scholar]
  106. De Simone, V.; Di Pasquale, V.; Giubileo, V.; Miranda, S. Human–robot collaboration: An analysis of worker’s performance. In Procedia Computer Science, Proceedings of 3rd International Conference on Industry 4.0 and Smart Manufacturing, Linz, Austria, 2–4 November 2022; Elsevier: Amsterdam, The Netherlands, 2022; Volume 200, pp. 1540–1549. [Google Scholar]
  107. Suta, P.; Lan, X.; Wu, B.; Mongkolnam, P.; Chan, J. An overview of machine learning in chatbots. Int. J. Mech. Eng. Robot. Res. 2020, 9, 502–510. [Google Scholar] [CrossRef]
  108. AlSulaimani, A.M.; Majumdar, P.; AlHammadi, R.; AlHammadi, B. Centralized Predictive Analytics & Diagnostics Value Creation. In Proceedings of the ADIPEC, Abu Dhabi, United Arab Emirates, 31 October–3 November 2022. [Google Scholar]
  109. Pokuta, W.; Zatwarnicki, K. Application of Neural Networks in Distribution of the Load in Cluster–Based Web Systems. Appl. Sci. 2022, 12, 79. [Google Scholar] [CrossRef]
  110. Borzemski, L.; Zatwarnicka, A.; Zatwarnicki, K. Global adaptive request distribution with broker. In Proceedings of the International Conference on Knowledge–Based and Intelligent Information and Engineering Systems, Vietri sul Mare, Italy, 12 September–14 November 2007; pp. 271–278. [Google Scholar]
  111. Ding, H.; Gao, R.X.; Isaksson, A.J.; Landers, R.G.; Parisini, T.; Yuan, Y. State of AI-based monitoring in smart manufacturing and introduction to focused section. IEEE/ASME Trans. Mechatronics 2020, 25, 2143–2154. [Google Scholar] [CrossRef]
  112. Dash, R.; McMurtrey, M.; Rebman, C.; Kar, U.K. Application of artificial intelligence in automation of supply chain management. J. Strateg. Innov. Sustain. 2019, 14, 43–53. [Google Scholar]
  113. Zatwarnicki, K.; Pokuta, W.; Bryniarska, A.; Zatwarnicka, A.; Metelski, A.; Piotrowska, E. General Environment Description Language. Appl. Sci. 2021, 11, 740. [Google Scholar] [CrossRef]
  114. Bryniarska, A. The model of possible web data retrieval. In Proceedings of the 2015 IEEE 2nd International Conference on Cybernetics (CYBCONF), Gdynia, Poland, 24–26 June 2015; pp. 348–353. [Google Scholar]
  115. Dale, R. GPT-3: What is it good for? Nat. Lang. Eng. 2021, 27, 113–118. [Google Scholar] [CrossRef]
Sensors 23 03826 g001 550
Figure 1. Main components diagram of the Hybrid Intelligence System (HINT).
Figure 1. Main components diagram of the Hybrid Intelligence System (HINT).
Sensors 23 03826 g001
Sensors 23 03826 g002 550
Figure 2. Graphic explanation of the idea of reflecting the process as an analogy to the towers of Hanoi presented in [64].
Figure 2. Graphic explanation of the idea of reflecting the process as an analogy to the towers of Hanoi presented in [64].
Sensors 23 03826 g002
Sensors 23 03826 g003 550
Figure 3. (A) Passive and (B) Active Quality Control in the exemplary implementation of the HINT system in a wooden joinery production industry company.
Figure 3. (A) Passive and (B) Active Quality Control in the exemplary implementation of the HINT system in a wooden joinery production industry company.
Sensors 23 03826 g003
Sensors 23 03826 g004 550
Figure 4. The co-bot interactive helper workbench. The interaction interface hardware is the same as presented in the next figure.
Figure 4. The co-bot interactive helper workbench. The interaction interface hardware is the same as presented in the next figure.
Sensors 23 03826 g004
Sensors 23 03826 g005 550
Figure 5. Interactive assisted assembly stand.
Figure 5. Interactive assisted assembly stand.
Sensors 23 03826 g005
Sensors 23 03826 g006 550
Figure 6. The employee’s last resort—a videocall with the expert (the expert’s workplace and communication application interface).
Figure 6. The employee’s last resort—a videocall with the expert (the expert’s workplace and communication application interface).
Sensors 23 03826 g006
Sensors 23 03826 g007 550
Figure 7. The adaptive matching mechanism of the presented content to the detected user interaction model: (A) before optimizations, (B) after optimizations.
Figure 7. The adaptive matching mechanism of the presented content to the detected user interaction model: (A) before optimizations, (B) after optimizations.
Sensors 23 03826 g007
Sensors 23 03826 g008 550
Figure 8. Number of meetings or video calls with the expert for individual participants. Blue bars—without the use of the HINT system, only paper documentation (information channels: text, overview photos) or physical contact with an expert. Red bars—with the use of the HINT system (information channels: text, overview photos, voice messages, overview video, remote contact with an expert). Data arranged in ascending order.
Figure 8. Number of meetings or video calls with the expert for individual participants. Blue bars—without the use of the HINT system, only paper documentation (information channels: text, overview photos) or physical contact with an expert. Red bars—with the use of the HINT system (information channels: text, overview photos, voice messages, overview video, remote contact with an expert). Data arranged in ascending order.
Sensors 23 03826 g008
Sensors 23 03826 g009 550
Figure 9. The number of changes of information channels depending on the advancement of the window assembly process using the HINT system).
Figure 9. The number of changes of information channels depending on the advancement of the window assembly process using the HINT system).
Sensors 23 03826 g009
Table
Table 1. Summary of the number of the information channel changes by the assembler.
Table 1. Summary of the number of the information channel changes by the assembler.
ParticipantImages (Graphical) [No. of Uses]Videos (Multimedia) [No. of Uses]Audio (Speech) [No. of Uses]Channel Replay [No. of Repetitions]Avatar Teleconference [No. of Uses]
144421
200102
300101
423220
500001
601501
700001
802110
937100
1021100
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Gardecki, A.; Rut, J.; Klin, B.; Podpora, M.; Beniak, R. Implementation of a Hybrid Intelligence System Enabling the Effectiveness Assessment of Interaction Channels Use in HMI. Sensors 2023, 23, 3826. https://doi.org/10.3390/s23083826

AMA Style

Gardecki A, Rut J, Klin B, Podpora M, Beniak R. Implementation of a Hybrid Intelligence System Enabling the Effectiveness Assessment of Interaction Channels Use in HMI. Sensors. 2023; 23(8):3826. https://doi.org/10.3390/s23083826

Chicago/Turabian Style

Gardecki, Arkadiusz, Joanna Rut, Bartlomiej Klin, Michal Podpora, and Ryszard Beniak. 2023. "Implementation of a Hybrid Intelligence System Enabling the Effectiveness Assessment of Interaction Channels Use in HMI" Sensors 23, no. 8: 3826. https://doi.org/10.3390/s23083826

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop