Resuming Experiences in Human-Centered Design of Computer-Assisted Knowledge Work Processes ^†

Abstract

Under labels like »AI« or »Machine Learning«, adaptive systems using methods of function approximation for adapting their performance to given sets of data from the environment are increasingly being deployed in the domain of knowledge work. Their design and effective use raise new questions with respect to their specific qualities. Summarizing relevant experiences from more than four decades of human-centered design of software artifacts and computer-assisted work processes, lessons learned are being reflected on with respect to what new challenges they bring about, and what new research questions they raise, respectively.

1. Multi-Agent systems

In recent times, so called »intelligent agent systems« [1] have been of much interest to industry and research. This paper addresses some practically relevant sociotechnical design aspects of agent systems that have been hardly treated so far. These kinds of automatically operating systems actually are adaptive systems, since their operating mode is—although controlled by programs—adapted to data from outside. Mathematically, it is a matter of functional approximation.

As adaptive systems, agents can receive sensor signals from the physical environment and interfere with it through actuators. They can send data to other technical systems, and have their own human-machine interface. They work in a fully auto-operational manner.

Although mistakenly called »intelligent« or even »autonomous«, they are actually not self-determined but other-directed by their designers. Thus, like any other computer system, they objectify specific aspects of cognitive work in programs. They are in no way more intelligent than other systems, since their operations are equally program-controlled by algorithms.

With these features, auto-operational agents can be mutually connected into »multi-agent systems (MAS)«. The MAS behavior then decidedly depends on their interference with the dynamically changing environment. Hence, it depends on history and formally equals the behavior of a so called »non-trivial machine« (Förster 1993, cf. Figure 1) with a double dependence of both the internal state and the output functions, on the input and the precedent internal state.

Due to the interference with the environment’s changing conditions and, hence, the dependence on history—the MAS behavior is more or less non-transparent and incomprehensible for users. In other words: the operational behavior of MAS is strongly situated and history-dependent, and therefore practically unforeseeable for users. This predicament has called for some research efforts towards designing »explainable artificial intelligence systems«, research which is lacking encouraging achievements so far [4].

2. Relevant Findings from Sociotechnical Systems Research

Although multi-agent systems are on the rise, it is rather unlikely that they will fully replace skilled knowledge workers—contrary to opposing, but illusionary expectations. Hence, we can draw on findings from earlier sociotechnical systems (STS) research and design methods that have been pushed, to a great extent, for designing computer-supported knowledge work since the 1980s. In particular, they have profited from the debates about and the failures of the knowledge-based systems approach to CIM (Computer-Integrated Manufacturing), emphasizing the relevance of the human working capacity’s »tacit dimension«.

Instead of trying to replace human skills with machines, STS design focuses on work tasks as the basic item for which assisting technical artifacts and organizational schemes are being designed—such that the resulting work processes foster human working capacity and well-being, as well as economic performance, of course: That is the meaning of the triangle of humans interacting with technology and organization (H-(T-O)) dealing with work tasks at the center within a context of social practices [5,6].

It is important to notice here that all these activities underlie a trade-off between physical tractability and social desirability, and both depend on the prevailing interests and power relations involved.

Intensive research and rich experiences from practical implementations over decades have created a wealth of knowledge for STS design. In the given context, some basic findings on qualities of good human work are particularly relevant: good work [6] must

• be practicable and reasonable,
• not do any harm or impair well-being and,
• particularly enhance the human working capacity—as an epitome of experience, capabilities and learning—as a specific human strength to be developed.

Moreover, as computer systems massively intervene in social practices of work, this makes STS design a highly self-referential endeavor by changing the very subject it is envisaging. That is also the reason why evolutionary and participatory design procedures are needed [7]. Moreover, due to the auto-operational nature of adaptive systems, collaboration with humans has the character of co-action rather than instrumental interaction. This implicates a number of new challenges for STS design.

3. Contradictory Work Demands Produce Mental Stress

Intentionally acting humans co-acting with adaptive systems then take the place of familiar instrumental human-computer interaction based on humans’ sufficient appropriation of the systems’ functions. However, human intentional and goal-oriented interaction with adaptive systems, i.e., their instrumental use, is no longer possible now due to the non-transparent situated system behavior.

Consequently, the new form of working with adaptive systems confronts skilled knowledge workers with a number of new challenges and tensions.

• High demands for coping with given complex tasks are persistently disturbed by uncertain system reactions.
• It is difficult or impossible for the workers to retrace and, hence, to understand the systems’ situated reactions.
• Workers are thus hindered from learning from experience, to sufficiently appropriate the system’s functionalities and to enhance their skills.
• Frequently, workers will be made accountable for failure and the damage resulting from it, despite loss of control.
• These uncomfortable situations regularly submit workers to pressures of contradictory work demands causing mental stress or even disorders.

From previous research, we have a proven, resource-based relational model that can explain the generation of mental stress by contradictory work demands.

Work demands and working conditions such as performance or appropriation conditions, work-related goals, subjective norms and values or social context conditions can—depending on their peculiarity—either result positively in an experience of self-efficacy or success, with a gain of competence and well-being, or they can act negatively as straining constraints, with stress reactions like discontent, chronic fatigue, retraction etc. [8,9] (cf. Figure 2).

According to this stress generation model, relevant examples of highly stressing working situations are:

• Contradictions between tasks and executing conditions restrain action regulation and learning options due to inadequate tools causing additional efforts.
• Conflict between tasks and learning conditions hinder workers from obtaining the appropriate necessary knowledge and the artifact’s technical functions.
• Contradicting project objectives put workers in »double loyalty« conflicts between different but equally important expectations to act.
• Contradictions between work-related and individual values put workers in conflict between project objectives and professional behavior or standards.

In the working situation at hand, where knowledge workers co-act with adaptive systems, they have to cope with the challenges [10] of:

• non-transparent systems’ behavior,
• hardly explainable systems’ behavior,
• uncertain results due to biased and unratable input data,
• misguided or unrealistic expectations of systems performance.

Knowledge workers trying to make effective use of these systems are then

• compelled to blindly trust in the systems’ outcome without any chance for their own assessment;
• being hindered from fully appropriating the systems’ functions for unrestricted instrumental use violating the basic »expectation conformity« requirement (EN ISO 9241-11);
• often being left in the dark about who is accountable for possible failures.

This has at least two prevailing problematic consequences:

Being submitted to such contradicting work demands and being under pressure of effectively accomplishing their tasks—without full control of their means of work—knowledge workers are at high risk of suffering from mental disorders [8] (cf. already [11]).

Well known »ironies of automation« [12,13] get even worse. Specifically, sufficient experience cannot be developed under normal conditions, while the knowledge workers’ vanishing competence under such conditions is still needed in case of system failure—which, due to non-transparent behavior, is even hardly detectible.

4. Conclusions

Informed by sociotechnical design knowledge and findings from the operational model of mental stress generation outlined, a number of conclusions for work design including co-action with adaptive systems can be drawn. In order to create productive and sustainable future work assisted by advanced computer systems, the following requirements need to be satisfied:

• In order to avoid contradictory demands, the implementation and use of adaptive systems should be concentrated on tasks that can reasonably be fully automated—the results of which, however, need to be blindly trusted.
• In operation with skilled knowledge workers, the use of adaptive systems should be avoided for the reasons explicated—as long as self-explaining systems are missing.
• Research efforts for adaptive systems should be concentrated on the development and implementation of facilities to explain questionable system behavior on demand.
• Like other high-risk technologies, adaptive systems should be submitted to publicly controlled certification procedures before deployment.
• In cases of failure, accountability regulations and practices need to be based on comprehensive scrutiny rather than simply ascribing them to human error.
• Consequently, capacities for comprehensively scrutinizing system failures need to be implemented