Sustaining Digital Adaptation ^†

1. Introduction

In response to “Digital Darwinism”, people, organisations and society need to adapt to the different characteristics of digital information. Successful adaptation requires an understanding of the nature of information and the information ecosystems (just “ecosystems” when the context is clear) that develop under a range of selection processes. Ecosystems develop conventions that determine the pace, quality and friction with which information is processed. These conventions are embedded in the structure and communications of ecosystems and the entities that comprise them; we call them Interacting Entities (IEs)—these include people, computer systems, animals, organisations or parts of organisations.

Ecosystem inertia means that these ecosystem conventions cannot often respond fast enough to digital change. Digital information offers the potential of faster pace and reduced friction, but to achieve these, and also build in the necessary information quality, ecosystems may require extensive change to respond to different and diverse interactions.

To make these changes sustainable, they need to be compatible with the changing selection pressures resulting from digital change. IEs need to be dextrous to respond fast enough to the environment (the term “dexterous” avoids confusion with the specific meaning of the term “agile” in the example used below). This paper considers the impact of digital change using an analysis of the structure and flexibility of IEs in information ecosystems and shows how to sustain digital adaptation. The ideas are applied throughout the paper to an IT department (as an IE) to highlight recent ideas about software and systems engineering (Agile and DevOps—so-called because it forges a much closer relationship between Development and Operations) in a digital environment.

2. Selection and Information Ecosystems

IEs interact with their environment through channels—they sense the environment and decide how to act. The interactions of an IE and their impact on outcomes determine the ability of the IE to thrive (or not)—they define the selection pressures on the IE. Favourable outcomes will enable IEs to create derived IEs (e.g., children, updated software product versions, changed organisations) through some development process. In this way, interactions determine how IEs evolve in ecosystems. The activity of other IEs (e.g., competitors) impacts the interactions available to an IE (and vice versa) especially when resources are scarce.

Information enables IEs to link the current environment state with potential outcomes and the actions potentially able to provide these outcomes. So, IEs deal with descriptive, predictive and prescriptive information and the connections between them.

Information ecosystems develop their own conventions for sensing, processing and acting using these different types of information. These conventions support different ranges of information measures (pace, friction, quality) that result from trade-offs driven by the selection pressures.

Complex IEs include different components (IEs in themselves) that exhibit different conventions. For example, businesses have Sales and Marketing, Finance, HR and IT departments. Each contains a mix of the ecosystem conventions of the business and those of the particular discipline. Each may contain smaller components; for example, an IT department may contain separate Development and Operations components. Selection pressures may apply differently to the different components of an IE.

We can categorise the selection pressures on IT organisations in the following terms:

• business selection pressures (“quality” in terms of the alignment of IT with business strategy);
• user selection pressures (“quality” in terms of system and service quality and reliability);
• efficiency selection pressure (“friction”);
• responsiveness selection pressure (“pace”).

If an IT organisation does not deliverable favourable outcomes for its organisation in response to these pressures then there may be a range of implications. The business may not be able to deliver its strategy or operations efficiently and effectively. For the IT department, people may lose their jobs, the department may be reorganised, budgets may be changed or “shadow IT” (IT managed from outside the IT department) may spread.

3. IE Patterns and Components

The success (or not) of an IE is influenced by its pattern and the ecosystem conventions it instantiates. This includes the components it has, how they are structured and their ability to support different types of interaction.

There are two important dexterity measures: flexibility and extensibility. Flexibility is measured in terms of the number of different interaction types that a pattern can support without change. Extensibility measures what level of resource (e.g., money) is needed to support other interaction types—how easy it is to change the IE.

Patterns for organisations are described in terms of an operating model that defines facets like: interaction channels, governance bodies (decision-making groups of people), performance management, processes, organisation design (structure and roles) and culture. All of these also set the internal selection pressures on staff and suppliers and they define how effectively the external selection pressures are translated into internal selection pressures—a mismatch between the two is a source of poor performance.

The interaction channels determine which others IEs an organisation can interact with and how. They determine how effectively an organisation can sense the environment and act. The flexibility of an interaction channel is determined by its reach and whether the IEs at the other end use it. The requirement to create new channels (e.g., a new web site) causes inertia. Flexility is supported by the ability to change interactions easily as part of existing processes (e.g., to post new content rather than re-create a web site).

Governance bodies have a fundamental role in delivering dexterity because they decide much of what will be done (e.g., financial approval to proceed with changes). The degree to which they do so depends on their terms of reference, the information available to them (and the degree to which it captures friction, quality and pace), the attendees and the frequency with which they meet.

The degree to which performance management supports flexibility is determined by the objectives defined, how they are assessed and the degree to which they reflect the selection pressures. Performance management can drive unanticipated behaviours when people “game” the measures or when the measures do not reflect the external selection pressures. Matching external selection pressures accurately is difficult to achieve, especially as they change. Performance management can reinforce discipline ecosystem conventions at the expense of external selection pressures. And, compared with the minute-to-minute impact of culture, performance management can be a blunt, infrequent tool to create internal selection pressures.

An organisation design aligned with functions (e.g., Development, Operations) can prioritise discipline ecosystem conventions over the market. By contrast, market orientation can (without care) ignore elements of good functional practice. However, with a functional design it is more difficult to align internal selection pressures with external selection pressures.

Processes support flexibility only if they are designed to do so and the implementation accurately reflects the design. Anyone who has called a poor quality call centre will understand the impact of inflexible processes and the constraints imposed by underlying systems. Process flexibility often demands flexible access to information rather than the limitations imposed by many systems.

Culture is the often unrecognised driver of success because it operates continuously (unlike many of the other facets) and so it provides ever-present selection pressures. It has been defined as “The specific collection of values and norms that are shared by people and groups in an organization and that control the way they interact with each other.” In other words it is at the core of ecosystem conventions in organisations. However, it can be difficult to change.

We can summarise these points in the following way: the facets create inertia when they are driven by friction and quality at the expense of pace. Often the facets are defined infrequently and changed only in response to major problems (often in a way that decreases pace). But culture is an insidious presence and it can determine the extent to which the others will enable dexterity.

For IT organisations, the prevailing operating models and conventions have been defined by the implementation of IT Infrastructure Library (ITIL) processes and phases of outsourcing, amongst other factors. As conventions have developed over decades, many issues have developed that indicate a mismatch in external and internal selection pressures, for example:

• unnecessary working practices developed in response to particular events and never discarded in the light of changed circumstances;
• “watermelon” performance—green (good performance) on the outside, according to supplier reports perhaps, but red (poor performance) on the inside as perceived by business users;
• missing interactions—in which the IT organisation does not interact sufficiently on (for example) strategic questions;
• mismatched measures—in which pace and quality were traded-off against perceived reductions in friction (ie cost-driven) or quality was traded off against perceived pace (ie delivery on time whatever the consequences);
• retreat into silos—in which selection pressures have focused separately on individual components rather the whole, respecting the discipline ecosystem rather than that of the organisation as a whole and causing difficulties and gaps between components.

4. Digital Information

“Digital Darwinism” is caused by fundamental changes to external selection processes. Digital information enables the same outcomes to be achieved in new ways with reduced friction and also enables new outcomes to be achieved; it stimulates a faster rate of change in the environment and a consequent requirement for IEs to increase their responsiveness. Increasingly, digital technology (including examples like machine learning) enables new approaches to information quality.

Digital technology has created numerous new channels and sources of information (e.g., the world-wide web, social media) through new devices. Digital information supports different types of information—initially the focus was on descriptive information (e.g., customer address) but increasingly also predictive information (e.g., propensity to buy) and prescriptive information (e.g., automated action).

Under the influence of Moore’s Law, digital technology has driven down friction by several orders of magnitude. As friction has been driven down, so increases in pace have been enabled—it has been possible to interact much more frequently and more responsively.

Quality was one of the initial casualties of reduced friction. Quality is too hard to assess routinely unless there is a very good reason so “quality by proxy” is the norm. Such norms have not yet been established for many new channels and quality has been assumed rather than proven. However, machine learning, amongst other approaches, can improve quality in particular domains by enabling types of prediction not available to people (relying, for example, on the analysis of very large amounts of data).

Sensing the environment has also changed dramatically. A proliferation of channels and information volume (“big data”) without understood levels of quality provides a new challenge in terms of measuring meaningful events in the environment (i.e., determining signal to noise). However access to so much data, with new tools (e.g., sentiment analysis), can provide much faster understanding.

In response to digital change, many organisations suffer from inertia and can no longer respond quickly enough. Recently, improving pace—the ability to respond fast and accurately to the environment—has been the focus for many organisations as part of their digital transformation.

5. Change and Dexterity

The challenge for organisations is to become more dextrous—with a pattern that makes them responsive to change. Flexibility and extensibility are key measures of dexterity. But the nature of the change needs to be clearly understood—it is important to sense the environment and determine the external selection pressures with sufficient certainty to be able to define the required internal selection pressures and operating model required.

IT has traditionally applied internal selection processes based on traditional engineering practices. But these were not designed to deal with the pace of digital change. Digital Darwinism requires:

• a short cycle time—the ability to turn new requirements into IT services fast;
• the ability to deliver many releases in a shorter time.

Established ecosystem conventions are not compatible with these requirements and new ways of working—Agile and DevOps—have developed specifically to meet the challenge of overcoming the conventions. They use automation but also apply lean ways of working originally developed by Toyota and now used extensively in services as well as manufacturing. Lean ways of working embed principles that link internal selection pressures and external selection pressures.

6. Changing the Pattern Sustainably

Changing organisations is not easy. The impact of ecosystem conventions is that old ways of working are deeply embedded in all aspects of the operating model, stifling change. In making a change there are two factors to consider: what pattern is required and how can internal selection pressures be encouraged to stay aligned with external selection pressures as they change?

The new pattern may require new components and, to overcome the ecosystem differences of components, new relationships between them. Each component may also require a new mix of the ecosystem conventions of the business and that of the particular discipline; or entirely new ecosystem conventions may need to develop. In particular, it is important to identify the elements of the operating model that will inhibit the change.

How would these ideas apply to an IT department? Each of the facets of the operating model may need to change overall and with respect to the different components. The Development and Operations components, often with different cultures and practices, present one of the largest challenges.

The implementation of lean techniques matches the internal selection pressures used within the IT department with the external selection pressures and applies them at a much more detailed scale. As the name implies DevOps is about Development and Operations departments working closely together despite their different cultures; DevOps has been defined as the application of lean ways of working to IT, supported by automation. Lean ways of working are an important factor making the cultural changes needed. DevOps builds on the following lean principles:

• the “voice of the customer”—in response to the business and user (“quality”) selection pressures;
• continuous removal of waste—in response to the efficiency (“friction”) selection pressure;
• flow and value stream management—in response to the responsiveness (“pace”) selection pressure;
• culture—combining responses to the selection pressures in day-to-day interactions with people rather than intermittently.

Combining these with automation (to support development, integration, testing and deployment) can provide large changes in both pace and quality and maintain the alignment of internal and external selection pressures.