Development of a Change Management Instrument for the Implementation of Technologies ^†

Abstract

The manufacturing industry is increasingly being dominated by information and communication technology, leading to the development of cyber-physical systems. Most existing frameworks on the assessment of such technological advancements see the technology as a solitary system. However, research has shown that other environmental factors like organizational processes or human factors are also affected. Drawing on the sociotechnical systems approach, future technologies could be evaluated using scenarios of digitized work. These scenarios can help classify new technologies and uncover their advantages and constraints in order to provide guidance for the digital development of organizations. We developed an instrument for evaluating scenarios of digitized work on the relevant dimensions ‘technology’, ‘human’ and ‘organization’ and conducted a quantitative study applying this instrument on three different scenarios (N = 24 subject matter experts). Results show that our instrument is capable of measuring technological, human and organizational aspects of technology implementations and detecting differences in the scenarios under investigation. The instrument’s practical value is significant as it enables the user to compare and quantify scenarios and helps companies to decide which technology they should implement.

1. Introduction

The manufacturing industry is increasingly being dominated by information and communication technology [1,2]. This creates a multitude of possibilities for the design of technological systems and enables the leap from mechatronic systems to highly automated cyber-physical systems. These cyber-physical systems have a significant impact on virtually all processes of the industrial value chain, ranging from individual assistance in product engineering and manufacturing [3,4] over organizational process management [5] to large-scale energy systems such as the smart-grid architecture [6]. Advantages of cyber-physical systems include their interconnectivity, real-time capability, modularity and virtualization of physical processes. This means they can communicate autonomously with each other in real time, be adapted to fast-changing needs and record all data for troubleshooting and future reference [7]. However, a wide variety of possibilities makes it difficult for companies to choose one technological solution that fits their needs and helps them achieve their goals. A premature investment in the wrong technology might result in workers not using the technology, performance goals not being achieved, or the technological infrastructure not being compatible, among other things. It becomes apparent, that a sole focus on the technology is too narrow so that we rather suggest considering scenarios of digitized work. A scenario is defined as a generally understandable description of a possible situation which is based on a complex network of characteristics of influencing factors [8]. Thus, a scenario-based approach gives a deeper understanding of different technologies and highlights interdependencies of factors related to the successful technology implementation. Consequently, the aim of this study is to present an instrument that incorporates all relevant factors of technology scenarios and that companies can use to weigh advantages and disadvantages of technological systems. More specifically, we investigate if the instrument is able to detect significant differences between scenarios of digitized work.

1.1. Background

To this day, several approaches of guidelines towards the assessment of (technological) systems were proposed. Notable examples are, among others, the framework for the implementation of additive manufacturing [9], a framework for the integration of collaborative robots in advanced manufacturing systems [10] or research on evaluation criteria for digital assistance systems [11]. However, most of the current frameworks in the field of digitization and cyber-physical systems rely heavily on a purely technological approach, neglecting the interdependence of the technology with organizational and human factors in a scenario of digitized work. There are other approaches, such as frameworks on the assessment of human-machine systems [12] or frameworks for human-centred design [13] that include human-related and organizational factors but lack the transfer into a practical instrument and have therefore a limited practical value. An application of these frameworks often requires a deep analysis and understanding of the current (manufacturing) system, a process which can be time and resource intensive. In addition, the frameworks usually address specific issues and organizations often do not have a clear overview about the overall benefits of implementing new technologies and processes.

1.2. Need for a Sociotechnical Approach

The theoretical basis for our instrument is the sociotechnical systems approach. It claims that organizations consist of a social and a technical system which are interdependent. The technical system comprises for example tools, procedures and machines that are used in the work process, the social system contains the employees, their attitudes towards the organization and their social relationships [14]. Thus, when implementing a new technology, these changes have an impact on the employees, their work environment and organizational processes [15]. Therefore, in order to reduce the risk of unwanted consequences, the benefit assessment of a planned technology should include technological, human-related and organizational factors. A meta-analysis on change management from the early 1990s already showed that a joint consideration of technological, personnel-related and organizational aspects had a greater effect on financial (e.g., efficiency, error rate) and behavioural (e.g., turnover, absenteeism) performance indicators than changes with a sole focus on one of the dimensions [16]. Aside from being able to estimate a technology’s consequences in a straightforward way, our practical instrument can also support change management to encourage employee participation.

1.3. Benefits in the Change Management Process

Change management refers to approaches and actions that are necessary for the successful design and implementation of change [17]. To give an example, research on the digitization of governmental processes has shown that a wide array of benefits such as more efficient and higher-quality processes can be achieved through technology implementation but require a thorough identification, understanding and most of all involvement of the stakeholders (e.g., the technology users) [18]. In this context, the literature discusses a number of factors and strategies, for example top management support, participation of affected employees, training to prepare for the new processes, incorporation into the business plan and vision and testing of the planned technology [19,20]. Employee participation in the change process is among the most studied success factors. A literature review has found that it is related to higher acceptance of change, a better understanding of the change and its benefits, a greater involvement in implementing behavioural changes, an increased attachment to the organization and lower change-related stress, among other things [21].

There are three reasons why employee participation is beneficial for the success of a change process. First, having a say during a (change) process is strongly related to perceiving this process to be fair, which is in turn associated with commitment to change and change-supporting behaviour [22,23]. Second, employees who can have an influence during a change process, experience more autonomy. That means they feel that they can align the planned changes in a way that is compatible with their own values [24]. A recent study has found that the experience of autonomy mediates the relationship between participation in a change process and less change-related stress as well as positive behavioural and cognitive change attitudes [25]. Third, the active participation in the change process can be regarded as job crafting. Job crafting refers to actions taken by employees to make their own jobs more meaningful, engaging and satisfying [26]. It is positively related to job satisfaction, organizational commitment and performance [27,28].

Hence, in order to successfully implement a technology, employees need to participate in the process. Participation can be facilitated by letting the affected employees answer the questions from our instrument. Afterwards, the results should be presented to the employees and adaptations and further actions should be discussed together.

1.4. Aim and Contribution of Our Study

The aim of this study is to bridge the gap between theoretical frameworks and practitioners’ need for clear guidance in technology implementation. We developed an instrument to assess scenarios of digitized work on three dimensions: technology, human and organization (for more details see [29]). The instrument enables practitioners to reflect on consequences of the technology to be implemented and review possible alternative technologies. The current study focuses on the instrument’s ability to detect differences in the scenario characteristics. Therefore, we expect the following.

Hypothesis 1.

The scenario characteristics of the scenarios ‘mixed mock-up’, ‘conformity management supported by a graph database’ and ‘digital documentation of the production progress’ differ significantly on the three dimensions human, technology and organization.

In the following chapters, we first describe the instrument and its facets as well as the procedure of our study. Here, we also give a description of the scenarios that were the research objects of this study. Second, we report the results of our study and provide visualizations of the developed instrument. Lastly, we discuss our study’s implications and point to further research directions.

2. Materials and Methods

We developed an instrument for the assessment of scenarios of digitized work. First, we give an outline of the instrument development process and describe the instrument in detail. Second, we apply this instrument to three scenarios, which are described briefly. Lastly, we illustrate how we investigated the three scenarios by using the instrument and describe the study design, the sampling method and the procedure.

2.1. Instrument for the Assessment of Scenarios of Digitized Work Based on Sociotechnical Criteria

In the following, our current work concerning the relevant criteria for scenarios of digitized work will be described in detail. In a first step, the different criteria for the dimensions have been chosen based on thorough literature reviews and expert interviews. In a second step, we aimed to provide a means of quantification or categorization, ideally based on existing questionnaires that addressed the same or similar issues. For this reason, each criterion is represented by one or more factors, which cover the different core aspects of the issue. The current accumulation of criteria that is presented in this paper has been iteratively worked out. An overview of the criteria and the corresponding factors for all three dimensions—technology, human and organization—can be found in

Table 1. Overview of the different criteria and the corresponding factors for the dimensions human, technology and organization.

Criterion	Factors
Human Dimension
Prioritization of abilities	Cognitive Psychomotor Physical Sensory Social/interpersonal
Qualification of employees	Amount of effort Form of qualification
Autonomy/scope of action	Work scheduling Decision making Work methods
(Possibility) of social interaction	Social interaction at the workplace Interaction with external parties Interdependent work tasks Task independence
Holistic nature of the task	Influence of the work on others Holistic nature of the work Automatic performance feedback
Work diversity	Task variety Task complexity Amount of information processing Creativity Variety of skills Variety of tools
Technological Dimension
Technological maturity	Category of technological maturity (introduction, growth, maturity, decline)
Compatibility	Mobile interoperability Stationary interoperability Personalized services Time-based services Location-based services Platform independence
Means of (human) interaction	Information tags Visual codes Direct tactile interaction Direct textual interaction Acoustic interaction Gesture recognition
Flexibility	Compatibility Connectivity Modularity
Complexity	(Inter)connectivity of the components Degree of understanding and active participation Specialization of the technology
Invest	Price (€)
Effort of implementation	Implementation effort
Cost of maintenance	Overhead System maintenance Upgrades System management Employee training
Safety	Training of employees Supervision of the work environment Implementation of security technologies
Security	Training of employees Supervision of the work environment Implementation of security technologies
Organizational Dimension
Product development process	Stage of the product development process (strategic product planning, product development, service development, production system development)
Promotion of innovation	Product orientation Customer orientation
Reduction of time-to-market	Process control Process optimization Infrastructural changes Interdisciplinary collaboration
Labour expenditures	Number of different activities Complexity of activities Number of participants Training period Necessity of distributed persons
Need for competencies/expert knowledge	Technology-/data-oriented Process-/customer-oriented Infrastructure-/organization-oriented
Influence on process organization	Adjacent/related processes Organizational units Customers/suppliers
Degree of collaboration	Organizational structure Mutual management
Degree of blurring of boundaries of work	Decentralisation Working time flexibility Project flexibility
Agility	Customer orientation and participation Transparency Decentralisation Breaking up disciplinary boundaries

.

2.1.1. Classification and Rating of the Human Dimension

Concerning the classification of criteria for the human factors, one of the main premises of our work has been the fact that with the implementation of new technologies, the characteristics of work have fundamentally changed over the last years [30]. Given that the tasks at hand in the scenarios of digitized work have changed, there has been a shift in the competencies which are required to successfully complete these tasks [31]. For a classification of the required skills, we focused on research by Fleishman and Quaintance [32], who argued that specified tasks can be characterized by specific competency profiles. Although the categories were not initially tailored to digitized work environments we see the categorization fitting for giving an overview of the nature of the task(s) in the scenario at hand. Following Fleishman and Quaintance, five categories of abilities were used for rating the scenario: cognitive, psychomotor, physical, sensory and social/interpersonal abilities.

A criterion which goes in line with a shift in the prioritization of the abilities is that new work processes often require additional qualification of employees in order to carry out the tasks in a scenario, a factor which has been identified as a critical requirement for the successful implementation of technology [33]. Given that the use of a new technology might require a specific skill set or certain competencies, this criterion is important for organizations since it allows an estimation of the effort that a new technology might bring in terms of workforce development. As for the rating, we have chosen to rate the qualification by the amount of effort that is needed to qualify personnel for the given scenario. In addition, for each scenario, the form of the qualification such as single-day training or workshops over multiple days were specified.

Besides the requirements of the employees, a factor that is crucial for a successful implementation of new technologies is the way the work is designed. Research has found that work design in general and work design of technologies in particular can have a significant impact on individual factors such as autonomy and cognitive ability [34] or organizational factors of work such as group feedback [35]. To assess the requirements, efforts, uses and constraints of work design, the work-design questionnaire (WDQ) [36] has been found to be a valid means of rating the scenarios. However, given the scope of our work, we altered the dimensions of the questionnaire to fit our needs for two reasons. Firstly, using the entire scope of the WDQ would surpass our aim to develop an easily accessible tool for rating scenarios of digitized work. Secondly, the rating system of the WDQ is highly subjective, given that it was designed for employees to assess their own work environment. Since our aim is the development of an objective rating scale, some of the items were altered. One of the most important work criteria has found to be the scope of action or autonomy of the work, a criterion which has been linked to the acceptance of technological innovations in organizations [37]. In terms of rating the scope of action of a work environment, we have chosen to adapt the three factors work scheduling, that is, the freedom of individual time management, decision making and work methods, that is, freedom in the choice of how to obtain work results. The second criterion which has been adapted from the WDQ is the possibility of social interaction. Given that factors such as social support are critical factors of well-being at work [38] and can help in overcoming work overload [39], the assessment of the work environment should not neglect this. For the rating, the factors social interaction at the workplace, interaction with external parties, interdependent work tasks as well as task independence have been adapted from the social characteristics of the WDQ.

The next criterion that has been extracted from the literature for the assessment of digitized work scenarios is the holistic nature of the task. This refers to the degree to which the job can be seen as a whole with identifiable results [40]. Understanding the holistic nature of the task has been found to increase the attractiveness of a job [41], classifying this criterion as an important indicator of a scenario of digital work. The rating included the factors influence of the work on other people, holistic nature of the work and automatic performance feedback. Lastly, the criterion of work diversity has been identified for the development of the questionnaire. Work diversity includes the possibilities of using a range of skills, knowledge and competencies that can be brought in during work to reach the employees’ goals [38]. The factors which were included in the questionnaire, cover task variety, task complexity, the amount of information processing, creativity, the variety of skills as well as the variety of tools that are needed to complete the task.

2.1.2. Classification and Rating of the Technological Dimension

Concerning the technological dimension, one of the most important concepts proves to be the criterion of technological maturity [42]. The classification of the technology into one of the four categories of the implementation and usage (introduction, growth, maturity and decline) constitutes a simple but effective means for organizations in having a first and clear classification of the technology.

Another criterion is the compatibility of the technology, indicating its ability to share any type of information across any technology component throughout the organization [43]. In our approach, we chose to rate the different forms of compatibility measured by the following factors: interoperability with mobile devices, interoperability with stationary devices, personalized services, situational services, time-based services, location-based services as well as the possibility of using the technology with different platforms. While compatibility has been identified a key factor in the communication and integration of the technology with other technical devices and systems, the communication between the machine and its human operator is not included in this category. Therefore, the means of (human) interaction with the technology is an additional factor in technology implementation. For the rating, we chose to assess the different means of interaction which can be used in the scenario, leading to the following factors: interaction with information tags (e.g., RFID), interaction through visual codes, direct tactile interaction, direct textual interaction, acoustic interaction as well as gesture recognition.

Another factor we included is the flexibility of the system. Research shows that flexibility of IT systems is a necessity for (rapid) changes in a business environment and the effective implementation of new technologies [44]. Concerning the quantification of flexibility, Duncan [43] identified three measurable key factors: The compatibility (the ability to share any type of information across any technology), the connectivity (the ability of the technology to communicate with other components in- and outside of the organization) and the modularity (the ability of the technology and its components to be reconfigured). Furthermore, research led by Schmitt emphasizes that low technological complexity may be positively related to successful technology implementation [45]. Concerning the assessment of complexity, it has been proven difficult to apply a quantitative means of measurement to this concept, given that for the broad range of currently used technology and IT systems, a single definition of the system’s complexity has not yet been established [46]. In an attempt to grasp the concept of complexity, we identified the three factors (inter)connectivity of the components [47], degree of understanding and active participation [48] and specialization of the technology [47] for the assessment in our questionnaire.

Another important criterion is the monetary effort of technology implementation. Concerning the cost of a technology, we can distinguish between the one-time initial invest in the technology, the effort of implementation in terms of personnel as well as the cost of maintenance of the technology. While the initial invest was easily quantifiable, since every technology comes with a defined price, the effort of implementation was more difficult to grasp. Given that for each organization, the amount of available personnel, the cost of personnel as well as the competencies can differ, no clear monetary value can be given for this criterion. However, we adapted a scale used in research by Gruber and colleagues [49] who distinguished the effort of technology implementation in four different stages: no need for additional tools, need for simple installation and additional tools, need for implementing new (complex) system components, need for a restructuration of the system. For the costs of maintenance of a technology, we tried to assess the effort of maintenance which is needed concerning the factors overhead of the organization, system maintenance, upgrades, system management and employee training.

Lastly, we identified the safety & security of the technology. In this case, safety is defined as the protection of the environment and the user from an object. In terms of technologies, this includes occupational safety, electrostatic discharge protection and the labelling of potentially hazardous elements [50]. For the rating of the criterion, we adapted a classification used by Saleh [51], who identified three main factors of safety: training of employees, supervision of the work environment and implementation of security technologies. The security of the technology goes in line with the safety of the system. With a heavier focus on the physical workspace, security addresses the protection of an information technology from its environment. Several methods of quantitative measurement of IT security have been found in the literature [52]. Nevertheless, these measurements often require a deep and narrow understanding of the technology at hand as well as rather complex methods, making them unsuitable for being used in our questionnaire. Instead, we chose to use the factors training of employees, supervision of the work environment and implementation of security technologies by Saleh [51] and adapted them to the criterion of security.

2.1.3. Classification and Rating of the Organizational Dimension

In addition to the evaluation based on technological and human criteria, the organizational dimension is essential for the assessment of scenarios of digitized work. Significant for this dimension is the general categorization of the scenario in regard to its position in the product development process. The corresponding stages are strategic product planning, product development, service development and production system development [53].

For companies, the criterion promotion of innovation can be decisive for the introduction of a scenario of digitized work. In order to assess the benefits for the promotion of innovation, the scenario is analysed in terms of the factors product orientation and customer orientation [54]. Furthermore, we included the reduction of time-to-market as a criterion because it helps reduce costs and product development time. For the assessment of this category, we included the following factors that can contribute to a reduced time-to-market: process control, process optimization, infrastructural changes and strengthening of interdisciplinary collaborations [55].

Labour cost estimates play a central role in project management [56]. The success or failure of project work, for example within the scope of a technology implementation, is decisively dependent on them. Factors to estimate the criterion labour expenditures are the number of different activities and their complexity for execution, number of participants, training period and necessity of geographically distributed persons. Besides labour expenditures to execute scenarios of digitized work there is a need for specific competencies and expert knowledge. This criterion is required for both technological implementation and execution. The results of the Competency Development Study 2016 divides the competencies that are crucial for companies to implement digitization and interconnectedness into three areas: technology and data oriented (e.g., IT security, artificial intelligence, data science), process-/customer-oriented (e.g., process management, systems engineering, eCommerce) and infrastructure-/organization-oriented (e.g., network/database administration, protection of privacy, dealing with specific IT systems) [57].

An indicator for assessing the complexity of a technology implementation is its influence on the process organization. We identified the following three factors to describe the process organization: adjacent/related processes, organizational units and customers or suppliers.

The degree of collaboration describes the distribution of decision-making powers within an enterprise or its subunits. This can influence the company’s policy and strategy and is therefore important from an organizational point of view. Weaver classifies this in cooperation, coordination, collaboration and coexistence [58]. For our instrument, we used a qualitative appraisal of the two factors organizational structure and mutual management. A scenario’s degree of collaboration results from the interaction of these two factors leading to one of the four former mentioned ways of collaborating, for example, a scenario with very strict organizational structures and no mutual management results in coordination as opposed to a scenario with flat hierarchies and a high amount of mutual management, which results in collaboration. Similar to the degree of collaboration, the degree of the blurring of boundaries of work can influence the operational organization by dissolution of ‘traditional’ structures. Factors are decentralisation [59] working time flexibility and project flexibility [55]. Lastly, agility is becoming more and more important for product development in order to meet the rapidly changing requirements of customers and the system context. For the assessment of scenarios of digitized work, the influence on agility is therefore significant. For this criterion, the factors customer orientation and participation, transparency, breaking up disciplinary boundaries and decentralisation of decision making have been identified as measurable indicators for agile projects [60].

2.2. Scenarios Under Investigation

We assessed three different scenarios with the instrument described above. The scenarios each describe the technology, its functionalities and tasks that are supported by the technology. Next, the process while using the technology is described, beginning with the initial situation, different phases and a final situation. The users’ tasks during the different phases are described. In the following, we describe the three scenarios under investigation. In our study, we presented the scenarios in German and translated them into English for the purpose of this paper.

2.2.1. Mixed Mock-up

Short Description

While testing new assembly systems, cardboard installations (so-called mock-ups) are used. This enables the employees to contribute their implicit knowledge in interdisciplinary workshops. However, some machine parts are not available before the workshop begins and the system is tested.

Due to the connection of construction data in 3D and physical mock-up by means of augmented reality (AR), the construction status of products and appliances is up-to-date and information is readily available. Because of this, the conception of the production system can begin much earlier. This shortens the building and test phases of the mock-up significantly. The visual simulation of the workplaces becomes more realistic for the employees by showing different machine parts by means of the AR technology. The employees’ task is to join the parts together to a workpiece which are displayed via AR technology and to reflect if the installation of the assembly system can be optimized.

Process

Initial situation: A group of three to 20 interdisciplinary agents are located in a room with the necessary infrastructure (physical mock-up, AR hardware). Remote agents can connect themselves with the group via other devices.

Phase 1: The virtual prototype/parts are projected by means of an AR system onto the prototypical assembly workplace. The main agent gathers the virtual part and afterwards checks the removal process concerning ergonomics, efficiency and so forth. The other agents can track the process on additional terminals and discuss it.

Phase 2: The main agent assembles the virtual part and validates the assembly process. The other agents can track the process on additional terminals and discuss necessary changes of the assembly system.

Phase 3: The assembled component is handed over to the following assembly station. The tests of the removal and assembly processes of the parts is repeated for all stations of the assembly line, using the AR technology.

Final situation: The production system is designed and can be handed over to the department of system integration.

2.2.2. Conformity Management Supported by a Graph Database

Short Description

A graph database makes it possible to test products and materials regarding specific requirements even without the expert knowledge of the employees. On the one hand, products, including components and substances from an existing system (e.g., product data management) are added to a graph database by the employees themselves, on the other hand there are all relevant standards, guidelines and so forth.

For example, the system enables employees to answer customer inquiries about the approval of products with regard to the conformity relating to specific regional requirements, such as the automatic test of a maximally allowed amount of a certain substance. The automatic test allows the output of a document on qualified conformity management. The task of the responsible employees is to maintain the database and to check the designed components before starting production.

Process

Initial situation: It starts with a customer requesting information, whether the supplied and installed product meets the requirements of a specific country where the customer wants to sell his or her product. The graph database is filled with data about the organization’s products as well as relevant norms and so forth of the main markets of the customers.

Phase 1: An employee adds the specific standards, guidelines and so forth of the target country to the graph database. The employee transfers the customer’s request to the service system.

Phase 2: The system carries out an automated comparison of the product with the target market requirements and gives the employee an overview of the fulfilled or unfulfilled requirements including a list of the reasons (e.g., positive laboratory test or exceeding the concentration of lead).

Final situation: Positive case (the product meets all target market requirements): the employee initiates an automatic output in form of a PDF file with his digital signature and sends it to the customer to confirm the conformity. Negative case (the product does not meet all target market requirements): the employee starts a process to check the product with regard to the unfulfilled requirements.

2.2.3. Digital Documentation of the Production Progress

Short Description

The documentation of the production progress in a manufacture (batch size 1) is carried out individually according to the customer’s specifications. Digital documentation enables the paperless recording of the production status. In this case, the production process is individual for each product. This enables the customer to follow the progress of the production of his or her product in real time. In the case of proposed changes, the customer can contact the producer before every step and communicate the wanted changes (e.g., colour request before painting).

The employees have to confirm the completed steps on a mobile device using a digital signature. Further information and annotations can be realized for example via a ’Voice to Text’ interface. The use of the technology shortens the process of documentation of the production progress for employees.

Process

Initial situation: The production process as well as the documentation templates are defined and implemented in the application. The workspaces in the production are equipped with appropriate input/output devices and are interconnected. There is an inquiry from a customer about the production progress and the manufacturing process so far.

Phase 1: By accessing the database, the service employee can view the digitally documented progress and quality features of the product in real time. Optionally, there is a direct access for the customer. During production, the employees document the progress digitally. For this purpose, they sign off the completed work steps using digital signatures (for example by scanning their ID). Further quality features can be recorded in comments, for example via ’Voice to Text’ (e.g., measured values).

Final situation: The service employee initiates an automated output in form of a PDF file with the current data records and sends it to the customer.

2.3. Study Design

The study using a cross-sectional design was realized by means of an online questionnaire which was created with the online survey tool Survey Monkey. An invitation containing a link was sent to subject matter experts from the field of work psychology, technology and business via email. Due to work experience in interdisciplinary projects these experts have a holistic view over the relevant fields. The study was conducted in October 2018 in Germany. Twenty-four subject matter experts took part in the study. To examine the inter-rater agreement the intraclass correlation was calculated. Over all three dimensions the raters corresponded with ICC = 0.94. For the single dimensions the agreement was ICC = 0.95 for the human dimension, ICC = 0.96 for the technological dimension and ICC = 0.87 for the organizational dimension. According to Chicchetti [61], these values indicate an excellent agreement of the raters on all three dimensions.

In order to compare the scenarios, each participant was required to create a code or received a code in their invitation to take part in the study. The participants first read the first scenario and then rated it using the instrument which was programmed as an online survey. Next, they read the second scenario and proceeded as described above, rating all three scenarios. First, the human dimension was rated, followed by the technical and organizational dimensions. The questionnaire contained 104 items in total. All criteria were measured by scales containing one to sixteen items. The statistical analysis of the criterion compatibility was conducted by using a sum value of all items. For all other criteria the mean of all items was calculated. For an overview of the criteria, sample items, number of items and the scaling see

Table 2. Overview of used measures.

Human Dimension
Criterion	Sample Item	No. of Items	Scale
Prioritization of abilities	To what extent are the following capabilities used in the scenario?	5	4-point Likert scale
Qualification of employees	There is a need for qualification.	2	4/5-point Likert scale
Autonomy/scope of action	The work can be planned independently.	3	4-point Likert scale
(Possibility) of social interaction	The exchange with other people in the workplace is given.	4	4-point Likert scale
Holistic nature of the task	The result of the work has a great impact on other people.	3	4-point Likert scale
Work diversity	A lot of information is processed at work.	6	4-point Likert scale
Technological Dimension
Technological maturity	The technology used in the scenario is a ... (a) pacemaker, (b) key, (c) basic, (d) repressed technology.	1	4-point Likert scale
Compatibility	The system offers situational services.	7	Dichotomous scale
Means of (human) interaction	The system uses acoustic interaction (e.g., voice control).	6	Dichotomous scale
Flexibility	The technology used in the scenario is highly compatible.	3	4-point Likert scale
Complexity	The technology used in the scenario is very specific for each application.	3	4-point Likert scale
Invest	Please rate the (estimated) cost of the technology.	1	4-point Likert scale
Effort of implementation	The installation of the technology used in the scenario requires complex re-planning and conversion of a system.	1	4-point Likert scale
Cost of maintenance	Please evaluate the operating effort of the technology used in the scenario with regard to system maintenance.	5	4-point Likert scale
Safety	With regard to security, the technology used in the scenario requires constant monitoring and documentation of the working environment.	3	4-point Likert scale
Security	With regard to safety, the technology used in the scenario requires a high level of training and safety instruction of employees.	3	4-point Likert scale
Organizational Dimension
Product development process	Which product development process applies?	4	Dichotomous scale
Promotion of innovation	In the scenario described, product orientation is made possible in the company.	4	4-point Likert scale
Reduction of time-to-market	The described scenario reduces the time-to-market by improving process control.	4	4-point Likert scale
Labour expenditures	The scenario described must be carried out by many people.	5	4-point Likert scale
Need for competencies/expert knowledge	Which experts are required for data evaluation and analysis?	16	Dichotomous scale
Influence on process organization	The scenario described has effects on other processes.	3	4-point Likert scale
Degree of collaboration	The scenario described favours an open and flexible organizational structure.	2	4-point Likert scale
Degree of blurring of boundaries of work	The scenario described favours a decentralisation of the company organization.	4	4-point Likert scale
Agility	The scenario described favours customer orientation and participation in dynamic projects.	4	4-point Likert scale

. Data were analysed using SPSS (Version 24). Computer code (syntax) is available upon request.

3. Results

Table 3. Descriptive statistics and statistical values of the analysis of variance of all factors for the dimensions human, technology and organization.

	Mixed Mock-Up		Conformity Management Supported by a Graph Database		Digital Documentation of the Production Progress
Human Dimension	M	SD	M	SD	M	SD	F(2)	p	ηp2
Cognitive abilities	2.94	0.87	2.89	0.90	2.50	0.79	1.87	0.17	0.10
Psychomotor abilities	3.06	1.00	1.28	0.57	1.61	0.78	30.61	0.00	0.64
Physical abilities	1.94	0.73	1.17	0.38	1.28	0.46	17.20	0.00	0.50
Sensory abilities	2.90	0.83	1.56	0.92	2.11	0.83	21.93	0.00	0.56
Social abilities	2.84	1.19	2.11	0.74	1.89	0.94	4.69	0.02	0.21
Qualification of employees	2.63	0.90	2.79	1.08	2.47	0.96	0.84	0.44	0.04
Autonomy/scope of action	3.55	0.42	3.44	0.42	3.52	0.43	0.20	0.82	0.02
(Possibility) of social interaction	2.55	0.77	2.76	0.65	2.60	0.67	0.56	0.47	0.03
Holistic nature of the task	2.80	0.56	3.02	0.65	2.72	0.73	1.80	0.18	0.09
Work diversity	2.80	0.56	2.39	0.50	2.21	0.37	9.99	0.00	0.36
Technological dimension
Compatibility	3.94	1.48	3.12	1.50	4.71	1.36	5.82	0.01	0.27
Means of interaction	0.48	0.29	0.36	0.28	0.59	0.21	5.83	0.01	0.27
Flexibility	2.72	0.63	2.40	0.64	2.79	0.57	2.02	0.15	0.13
Complexity	2.68	0.63	2.67	0.70	2.65	0.65	0.02	0.99	0.00
Invest	2.74	0.73	2.32	1.06	2.68	0.95	2.00	0.15	0.10
Effort of implementation	2.84	0.90	2.53	1.26	3.11	0.88	2.04	0.14	0.10
Cost of maintenance	2.48	0.60	2.32	0.62	2.39	0.64	0.54	0.59	0.03
Safety	1.83	0.68	1.20	0.43	1.67	0.80	4.70	0.02	0.22
Security	2.11	0.78	2.46	0.98	2.47	0.78	1.22	0.31	0.06
Organizational dimension
Promotion of innovation	2.89	0.58	2.95	0.70	2.77	0.72	0.36	0.70	0.02
Reduction of time-to-market	3.10	0.57	2.50	0.57	3.00	0.57	9.27	0.00	0.35
Labour expenditures	2.20	0.61	1.74	0.55	1.92	0.63	3.17	0.05	0.15
Influence on process organization	2.82	0.83	2.98	0.79	3.25	0.66	2.10	0.14	0.11
Degree of collaboration	2.79	0.84	1.75	0.62	2.29	0.92	7.30	0.00	0.40
Degree of blurring of boundaries of work	2.48	0.70	2.19	0.78	2.43	0.74	1.50	0.24	0.10
Agility	3.17	0.65	2.82	0.74	3.10	0.67	1.49	0.24	0.09

presents the means and standard deviations of all criteria on the dimensions human, technology and organization separated for the three scenarios mixed mock-up (MMU), conformity management supported by a graph database (CMGD) and digital documentation of the production progress (DDPP). Furthermore, Table 3 shows findings of the analysis of variance that was conducted to compare the specific scenario data. On all three dimensions we found statistically significant differences between the scenarios on at least two criteria. These findings support our hypothesis that the scenario characteristics of the scenarios mixed mock-up, conformity management supported by a graph database and digital documentation of the production progress differ significantly on the three dimensions human, technology and organization. In the following we will present the detailed results for each dimension separately.

3.1. Results Human Dimension

On the human dimension we found significant differences between the three scenarios on the following criteria: psychomotor abilities (F(2) = 30.61; p = 0.00), physical abilities (F(2) = 17.20; p = 0.00), sensory abilities (F(2) = 21.93; p = 0.00), social abilities (F(2) = 4.69; p = 0.02) and work diversity (F(2) = 9.99; p = 0.00). As shown in Figure 1, all criteria were rated highest in the MMU scenario. Regarding the need for psychomotor abilities, the difference between the MMU scenario and the CMGD scenario (t(19) = 7.32, p = 0.00) and the DDPP scenario (t(18) = 6.30, p = 0.00) was significant. The same applied to the need for physical abilities, CMGD scenario (t(19) = 5.81, p = 0.00), DDPP scenario (t(18) = 4.44, p = 0.00). The rating for the need for sensory abilities also differed significantly between the MMU scenario and the CMGD scenario (t(19) = 7.26, p = 0.00) and the DDPP scenario (t(18) = 4.40, p = 0.00). Here, we also found that the need for sensory abilities was significantly higher in the DDPP scenario than in the CMGD scenario (t(17) = 2.56, p = 0.02). The need for social abilities was again rated significantly higher in the MMU scenario compared to the CMGD scenario (t(19) = 2.32, p = 0.03) and the DDPP scenario (t(19) = 2.26, p = 0.04). The same applied to the rating of work diversity, CMGD scenario (t(19) = 2.87, p = 0.01), DDPP scenario (t(19) = 5.24, p = 0.00).

3.2. Results Technological Dimension

On the technological dimension we found significant differences between the three scenarios on the following criteria: compatibility (F(2) = 5.82; p = 0.01), means of interaction (F(2) = 5.83; p = 0.01) and safety (F(2) = 4.70; p = 0.02). As shown in Figure 2a,b, compatibility and means of interaction were rated highest in the DDPP scenario. As can be seen in Figure 2c, safety was rated highest in the MMU scenario. The compatibility was rated significantly lower in the CMGD scenario than in the MMU (t(17) = −2.16, p = 0.05) and the DDPP (t(16) = −3.70, p = 0.00). The same applied to the criterion safety, MMU scenario (t(18) = −3.43, p = 0.00), DDPP scenario (t(18) = −2.55, p = 0.02). Means of interaction was rated significantly higher in the DDPP scenario than in the CMGD scenario (t(17) = 4.32, p = 0.00).

3.3. Results Organizational Dimension

On the organizational dimension we found significant differences between the three scenarios on the following criteria: reduction of time-to-market (F(2) = 9.27; p = 0.00) and degree of collaboration (F(2) = 7.30; p = 0.00). As shown in Figure 3, ratings on both criteria were highest in the MMU scenario. Reduction of time-to-market was rated significantly lower in the CMGD scenario than in the MMU scenario (t(17) = −3.98, p = 0.00) and the DDPP scenario (t(17) = −4.02, p = 0.00). Regarding the degree of collaboration, the difference between the MMU scenario and the CMGD scenario (t(19) = 4.62, p = 0.00) and the DDPP scenario (t(18) = 2.24, p = 0.04) was significant.

4. Discussion

In this article, we presented and tested an instrument for the assessment of technologies that incorporates criteria from the dimensions human, technology and organization. In an empirical study with subject matter experts as participants, we investigated if our instrument was capable of detecting differences between three distinct scenarios of digitized work. The results support our hypothesis and show that the ratings of a number of criteria differed between the three investigated technology scenarios.

Most existing frameworks for the assessment of technologies have a sole focus on technological criteria. Yet, our study illustrates that a sociotechnical approach is necessary because the scenarios of digitized work differed not only in technological criteria like compatibility but also in human-related criteria like need for sensory abilities and organizational criteria like reduction of time-to-market. Our results are in line with previous research that has found that change initiatives have a more positive outcome when technological, personnel-related and organizational aspects are jointly considered [11].

4.1. Limitations and Future Research

As the instrument is at an early stage of development and this is the first quantitative study applying it to scenarios of digitized work, it was not possible to draw conclusions concerning the validity of the instrument. Further research should investigate the instrument’s validity comprehensively by correlating it to relevant external criteria, such as job satisfaction or health (for the human dimension), increase in efficiency (for the technological dimension) and innovation (for the organizational dimension). Moreover, we suggest to also include change-related constructs such as change readiness, technology acceptance and technology affinity as these constructs are predictors of successful technology implementations [62,63]. For a comprehensive validation, our sample size is not sufficient; however, we were able to show that the instrument is able to detect differences with regard to different aspects of technologies.

Moreover, the scenarios were rated by subject matter experts rather than future users of the instruments. In contrast to users in companies, the subject matter experts had no compulsory aim of implementing these technologies at their workplace which does not correspond to the future use of the instrument. However, this is also an advantage as this allowed the experts to rate the scenarios in an un-biased way and independently. Also, this allowed us to select a variety of scenarios, as we were not restricted to specific aims.

In this study we used a within-subjects design. Although the order of scenario evaluation was not specified, the research data showed that almost all experts evaluated the scenarios in the same order. To exclude sequence effects we suggest choosing a between-subjects design in future studies.

In future, studies should be conducted in which our instrument is applied in real companies. The results support decision-making on which technology should be implemented. These application studies should also investigate the impact of the technology, ideally on pure technological outcomes, as well as on human and organizational aspects.

4.2. Practical Implications

Our instrument helps organizations to assess alternative technologies. Due to the fact that all relevant characteristics can be compared at a glance, it is easier to weigh the alternatives. For example, if one technology scenario was rated higher in terms of promotion of innovation but lower in agility, the organization could choose a technology based on which outcome was more important. Thus, the instrument facilitates and structures the decision making process.

Furthermore, the prospective nature of the instrument supports the work of the human resources department. Knowing about human-related consequences of a technology before its implementation gives the human resources department the chance to prepare employees for the upcoming changes. For example, they might need a specific training or their job tasks need to be restructured. Moreover, the instrument can be used to support change initiatives. By letting the affected employees fill out the questionnaire, they feel more involved in the change process and will more likely accept it.

4.3. Conclusions

Due to the fact that a growing number of work processes is supported by technologies like cyber-physical systems, it has become increasingly difficult for organizations to select one technology that helps them achieve their goals. Therefore, we have developed an easily applicable instrument for the assessment and comparison of alternative technology scenarios. Our study found that the instrument serves its purpose in detecting differences between distinct scenarios and is therefore a valuable tool during digital change endeavours.