1. Introduction
The process of sensemaking has been described as one of finding meaning and gaining insight from information [
1]. During sensemaking, we engage in a process of comprehension [
2] in which we gather, restructure and reorganise information to help build a plausible understanding about an aspect of the world. There are a number of theoretical accounts offered on sensemaking. For example, Klein
et al. [
2] propose the data-frame model of sensemaking; Weick [
1] describes an analysis of sensemaking in organisations; Russell
et al. offer the learning loop complex [
3]; Dervin offers a sense-making methodology [
4]; and Pirolli and Card describe an analysis of sensemaking by intelligence analysts [
5].
Representation is a key part of sensemaking: Klein
et al. [
2] describe our understanding of events or situations as frames. A frame is a representation that serves as an account of a given situation in the mind of the sensemaker and can be expressed in a number of forms, such as stories, maps, diagrams or scripts [
6]. Pirolli and Card [
5] describe the representation and re-representation of information as key to the generation of insight during sensemaking in intelligence analysis. Their model shows analysts re-organising information, gathered through information foraging, into structured forms (
i.e., schematisation) to support reasoning. As Pirolli and Card point out, sensemaking often involves the creation and use of external representations, embodied in external artefacts, such as maps or lists. These presumably help in tasks that might otherwise be difficult to perform in our minds alone.
In his work on distributed cognition, Hutchins argues that the study of human cognition must look beyond the head of the individual to include an entire “cognitive system”, where this embraces cognitive processes as distributed across the social interaction of individuals, the artefacts they use, the environment they work in and previous events and experiences [
7,
8,
9]. For current purposes, our interest is in cognition as distributed across representations embodied within physical artefacts. Extending the unit of analysis beyond what happens in the head makes good sense here given the role that external artefacts can play in sensemaking. External artefacts come with built-in affordances that people can exploit to support external representation or computation where they judge that these might improve sensemaking some way. The use of representations “in the world” changes sensemaking, ideally for the better. Commitment to this is clearly demonstrated in the amount of work in academia and industry aimed at developing software for supporting this aspect of sensemaking (e.g., tools for interactive visualisation, argument mapping, spatial hypertext to name a few).
However, though current theories of sensemaking recognise the role of external representations, they do not seem to address the role of external representations or external artefacts in very much depth [
10]. For example, Pirolli and Card [
5] touch on the use of external data sources and external resources in sensemaking, particularly during
schematization, where information is represented schematically, perhaps as written diagrams or as visualisations, such as timelines, in order to identify patterns and to build inferences. Although this highlights the importance of external representation to the sensemaking process, there currently exists no framework for describing and evaluating the features and semantic meaning imbued by such artefacts that play a role in sensemaking in current theories. The development of tools and technologies for the support of sensemaking activities requires a deep understanding of sensemaking and associated analytical processes [
11], which includes the role of external representations and the interaction of different elements of cognitive systems, in a range of task settings. This requires the study of working environments and practices to draw out constructs and analytical approaches, which might provide a theoretical “lens” for analysing sensemaking situations, carving out space to consider design alternatives. In our work on distributed sensemaking, we aim to address this.
In distributed sensemaking, we consider sensemaking through the lens of distributed cognition, with a particular interest in how different external representations are used to support human reasoning. In this paper, we develop these concepts through the study of military signals intelligence (SIGINT) analysts taking part in a training exercise. We present an analysis that develops a model of “distributed sensemaking”. Our model consists of two elements: inference trajectories and a number of levels of description. Inference trajectories allow us to model the interaction of information embodied within different representational states in sensemaking. We also consider artefacts at a number of levels of description by which we can analyse the properties and affordances of them that play a role in sensemaking.
We present our model as an extension to existing sensemaking theory, addressing a gap in the analysis of the features of external artefacts and representations that play a role in it. Our aim is to provide a construct that sensitises analysts into thinking about this.
Our paper is structured as follows: in the next section, we give some background on sensemaking and distributed cognition. In
Section 3, we describe a case-study of military intelligence analysis before we deliver a summary of the findings of the aforementioned study. In
Section 4, we present a model of distributed sensemaking consisting of inference trajectories and a number of levels of description. Finally, we conclude our paper in
Section 5.
3. Case Study of Military Analysis
3.1. Scenario and Study Design
We carried out two runs of a study in which we saw experienced SIGINT analysts taking part in an exercise developed for the training of real-world analysts within an intelligence cell. Our study was designed to be consistent and authentic with respect to real-world analysis. Two analysts took part, each with 24 and 34 years of operational military intelligence analysis, respectively. Our study also involved two SIGINT trainers who played supporting roles (direction finder and supervisor) in the exercise and were responsible for developing the training scenario, with 24 and 25 years of experience, respectively.
The training exercise focused on a scenario in which the south coast of England was under threat from an opposing force. Our attention was on the role of the analyst within a signals intelligence cell whose task it is to build a situation picture from intercepted communications.
A typical structure of a signals intelligence cell is illustrated in
Figure 1. Interceptors (far left) are radio operators in the field who intercept radio broadcasts and communications and feed information to the Direction Finder (middle left). This information includes radio frequencies and modes used to make communications, excerpts of communications that may contain military unit callsigns (aliases used when making radio communications, e.g., AB01), codewords (cover terms used to conceal sensitive information or commands), as well as the geographic location (longitude and latitude) of units. The Direction Finder’s (DF) job is to triangulate the positions of units in the field included in feeds from multiple interceptors and to compile a tactical tip off (TTO) report to be sent forward to the analyst.
A TTO is typed into a pre-formatted form using a word processor and contains locations of units and corresponding callsigns, radio frequencies and modes (e.g., FM, AM) that they are using, plus important (or deemed to be) extracts from communications, including codewords. The analyst relies primarily on such reports to build a picture of the situation on the ground.
The analyst uses various pieces of software to receive TTOs, record aspects of the situation picture (e.g., locations of units on a map and within a command structure) and write Intelligence Summary (INTSUM) reports, which are periodically sent forward to the supervisor.
The study was set up to observe the role of the analyst, since this seemed to be the primary sensemaking role in the cell. The roles of interceptors/direction finder and supervisor were simulated by the two military trainers who designed the training scenario. Two runs of the study were conducted with experienced analysts, who were unfamiliar with the exercise scenario. The duration of the studies were two hours and one and a half hours, respectively.
Analysts used four computer displays arranged in a two-by-two formation (see
Figure 2). The software used by the analysts included the EW Training and Mission Support Tool (EWMST) (EWMST is proprietary software developed by MASS Consultants Ltd. (Lincoln, UK)), which allows the visual mapping of units according to latitude and longitude and allows the addition of other properties, such as callsigns and radio frequency, as metadata for each unit. IBM’s i2 Analyst Notebook was used to create network graphs illustrating the opposing force’s command structure as inferred by the analyst. Microsoft Word was used to read TTO reports and to create INTSUM reports to be sent to the supervisor. Instant messaging software was used to communicate and send files between the analyst, DF and supervisor.
A set of “working aids” in the form of printed materials were given to the analyst. These contained a number of tables of data and relating information known about the opposing force, such as radio equipment and callsigns at different echelons in the military command structure (
i.e., a P-404 radio is used at a frequency range of 1.25 to 4.5 MHz to send communications from the regiment to battalion level; see
Figure 3), radio encryption information (known information about encryption systems used to encode messages at certain echelons within the command structure), codewords used and the enemy order of battle (ORBAT). An ORBAT is a representation drawn from prior intelligence that describes the known structure of a military force, including equipment used, the structure of military units and command hierarchies.
5. Conclusions
In this paper, we have presented a study of sensemaking in action, with an analytical focus on the role of representational artefacts in the sensemaking process. We described the study of two experienced ex-military signals intelligence analysts taking part in a training exercise, which was designed to train real-world military personnel. In our analysis of this study, we focused on the role of representational artefacts in the sensemaking process, considering the properties of different material representations and how these properties affect reasoning and sensemaking. In our findings, we saw how representations, namely a series of “working aids”, played a crucial role in the analysts’ sensemaking. We saw how abstract information was used in concert with external artefacts leading to the generation of inferences and a situation picture.
We presented findings from our study in the form of a conceptual model consisting of inference trajectories and a number of levels of description. Inference trajectories offer a way of decomposing sensemaking into a series of steps, each of which contributes to the generation of an increasingly more specific and more differentiated situation picture. Mapping out a trajectory in this way allows us to analyse the way in which information is extracted from representations, often embodied within physical artefacts, and combined through inference making to contribute to understanding. In the current example of SIGINT analysis, inference trajectories illustrate (see
Figure 4) how the conclusion drawn from one inference acts as a premise in subsequent inferences. They also draw attention to visual lookup and cognitive reasoning operations at each stage, inviting us to consider how the physical manifestation of represented information affects sensemaking.
Findings made by Wong and Kodagoda [
26] map out similar reasoning processes in criminal intelligence analysts, showing the relationship between claims and premises throughout an analysis. It is apparent that there are comparisons that can be made between their analysis of criminal intelligence analysis and our analysis of SIGINT analysis. Though their study looks at the abstract reasoning of analysts and not the features of the artefacts they use, we argue that the comparisons between both studies support the applicability of inference trajectories for mapping out distributed sensemaking in both structured tasks, such as SIGINT analysis, and unstructured tasks, such as criminal intelligence analysis.
We also introduce the concept of a situation picture, referring to a sensemaker’s current understanding of a given situation, which becomes better defined as sensemaking develops. A situation picture also serves as a springboard into action, motivating the sensemaker to seek out further information and insights in order to gain a clearer understanding of a situation. Similar findings were also made by Wong and Kodagoda [
26]. We saw situation pictures represented externally, in the form of an INTSUM report created by the analyst, which evolved throughout our study as the situation picture developed. This was also seen in a study by Andrews, Endert and North [
28], who saw analysts record a form of situation picture in electronic and written notes.
We present three levels of description for examining such representational artefacts, taking into account their physical, semantic and pragmatic properties. An artefact’s physical constitution can play a role in the way it can be used in the sensemaking process. For example, we described how a table of radio frequencies and corresponding information printed on paper afforded being manually crossed out by analysts, allowing them to narrow down possible levels of command. When describing the semantic properties of an artefact, we examine the representational meaning artefacts are imbued with, that is what aspect of the world they are taken to represent. At a pragmatic level, we describe how an artefact gains meaning in light of current cognitive activity, revealing how it is used by virtue of its physical and semantic properties.
By analysing the role of different representations and representational artefacts in the ways we have demonstrated here, we can isolate specific interactions with such artefacts that contribute to the sensemaking process in a systematic way, which current theories in sensemaking and distributed cognition cannot. For example, in a distributed cognition study of air traffic control, Fields and colleagues [
32] describe the co-ordination of multiple forms of representational media, such as “flight strips”, “strip boards” and radar read-outs in forming an understanding of the current state of airspace. They describe a number of properties of these cognitive artefacts, including the flexibility and accessibility of representations and the operations and actions that can be performed on them. In concluding their paper, however, the authors concede that their approach lacked a method that took them from analysis to design. Moreover, their analysis focuses on making isolated descriptions of the use of a number of representational artefacts, but does not describe their role in the whole picture in terms of the co-ordination of information and representations to reach an understanding of the current state of airspace. We believe our model can go some way in filling this gap.
For example, by creating inference trajectories, we can map out air traffic controller’s reasoning processes through time, decomposing the interaction and co-ordination of information and resources, such as aircraft metadata written on flight strips and airspeed indications from a radar screen, which lead to the generation of a situation picture. Furthermore, by describing the physical, semantic and pragmatic properties of such artefacts, we can reveal which features of them play a role in the sensemaking processes of the air traffic controllers. For instance, a physical flight strip affords the air traffic controller the ability to write aircraft metadata on it and place it in a strip board alongside other strips. The strip represents a single aircraft, and the strip board is a collection of strips representing all aircraft in a given section of airspace. By moving the position of strips on the strip board, the air traffic controller can represent the positions of the aircraft in relation to each other at a particular moment in time.
By mapping out inference trajectories and describing the physical, semantic and pragmatic properties of representations in this way, we provide a systematic mechanism by which we can generate a deep understanding of distributed sensemaking in such scenarios. Inference trajectories reveal the coordination and interaction of information embodied within different artefacts, which lead to inference generation and the development of a situation picture. By analysing the physical, semantic and pragmatic properties of artefacts, we can examine the features of artefacts that play a role in sensemaking. This allows us to consider properties of interactions, such as user-costs, effectiveness, propensity for error or other properties that may be important to the particular sensemaking activity, carving out a problem space to consider design interventions that might change these properties in more or less desirable ways.
As highlighted by Chin, Kuchar and Wolf [
11], the development of new technologies aimed at supporting sensemaking requires deeper understanding of the analytical processes intelligence analysts carry out, including the way they use information tools. Liu, Nersessian and Stasko [
25] also argue that the developments of visualisation tools requires an understanding of the role of external artefacts and representations, citing distributed cognition as providing a useful perspective in framing this. We believe our work extends these notions. Though here we mainly focus on the findings of one study, we believe that our model can be applied in a number of different settings. We have drawn on the findings of studies in more unstructured and explorative settings, such as criminal intelligence analysis, to demonstrate this, showing concurrence with our findings.
Looking ahead, our intention is to develop our model further by carrying out studies in a variety of work settings to develop and validate its concepts further, leading to a robust and generalised framework that can be used to carry out analyses of sensemaking scenarios.