Next Article in Journal
Development of a Forecasting Framework Based on Advanced Machine Learning Algorithms for Greenhouse Gas Emissions
Next Article in Special Issue
Introduction to the Special Issue on Systems Thinking and Strategic Management
Previous Article in Journal
Examining the Impact of Entrepreneurial Orientation, Self-Efficacy, and Perceived Business Performance on Managers’ Attitudes Towards AI and Its Adoption in Hospitality SMEs
Previous Article in Special Issue
System Dynamics Modeling: Technological Solution to Evaluating Cold-Chain Meat Packaging Scenarios
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Multi-Methodological Conceptual Framework to Explore Systemic Interventions

by
Alberto Paucar-Cáceres
1,* and
Leila Abuabara
2
1
Department for Operations, Technology, Events and Hospitality Management, Manchester Metropolitan University, Manchester M15 6BH, UK
2
Instituto Tecnológico de Aeronáutica (ITA), Universidade Federal de São Paulo (UNIFESP), São José dos Campos 12228-900, SP, Brazil
*
Author to whom correspondence should be addressed.
Systems 2024, 12(12), 527; https://doi.org/10.3390/systems12120527
Submission received: 25 September 2024 / Revised: 15 November 2024 / Accepted: 21 November 2024 / Published: 27 November 2024
(This article belongs to the Special Issue The Systems Thinking Approach to Strategic Management)

Abstract

:
The paper proposes a ‘Systemic Multi-methodological Framework’ for multi-methodological management science/operational research (MS/OR) interventions. Based on John Mingers’ framework for mapping MS/OR methodologies/methods/techniques, we advance a systemic framework to enhance systemic intervention. The framework draws from the key elements of Robert Flood and Mike Jackson’s and Mingers’ concepts for multi-methodological practice. We discuss both the practical difficulties of applying Mingers’ notional systems in the real world and the cultural and psychological obstacles that prevent the viability of a multi-method and multi-paradigm intervention. We discuss the insights that are useful for overcoming the obstacles inherent to a multi-methodological intervention. By proposing an original framework, we aim to contribute to the debate about increasing systemic interventions and multi-methodological practice in MS/OR.

1. Introduction

Multi-methodological practice in operational research (OR) has been widely used to combine hard and soft approaches in different settings around the world. Some examples of both multi-methodological frameworks and applications include [1,2,3,4,5,6,7,8,9,10,11,12,13], amongst others.
In this paper we focus on contributing to the theoretical debate about the feasibility of composing a multi-methodological framework and, in that way, offer a device that hopefully will enhance current multi-methodological MS/OR practice. Drawing from the work of Michael Jackson’s and John Mingers’ approaches to multi-methodology in operational research, we advance a multi-methodological framework. We propose to use some of its features to create a ‘Systemic Multi-methodological Framework’ (SMMF). The framework draws from the key elements of Flood and Jackson’s total system intervention (TSI), operationalised somehow in Jackson’s Systems of Systems Methodologies (SOSM) and Mingers and Brocklesby’s framework for mapping methodologies. Specifically, we aim to further enhance OR/MS systemic practice by incorporating into SMMF Jackson’s and Mingers’ concepts of the three notional systems, namely (i) the problem content system, (ii) the intellectual resources system, and (iii) the intervention system [14] (p. 295). We use these three notional systems to design and shape the intervention and to assess how to deploy a portfolio of OR methods when bringing improvement to a situation considered problematic.
It is worth mentioning that an initial version of the framework, together with an application to an Argentinian small and medium-sized enterprise (SME), has been reported in two earlier pieces of work [9], one of which was an article in Spanish, in [10]. The present article has now refined the theoretical framework used in these two previous versions. Although this time we do not report on an application, here, we propose a novel amalgamation of Jackson’s and Mingers’ concepts and ideas on multi-methodology in OR.
We deliberately focus on these authors because, since their insertion in the 1990s, approaches and applications of multi-methodology have been mostly driven by the ideas and concepts presented by these two authors. We then embark on exploring the possibility/feasibility of bringing these ideas together. In this sense, the main research questions that guide this study can be seen as:
  • is the amalgamation of Jackson’s critical systems thinking and Mingers’ critical pluralism possible?
  • does this amalgamation offer a way to enhance a systemic intervention?
This article is organised as follows. Beyond this introduction, in Section 2, we review and discuss the key features of multi-methodological practice proposed by Mingers’ multimethodology frameworks. In Section 3, the systemic multi-methodological framework (SMMF) to explore and reflect on a systemic intervention is presented, detailing its design and possible uses. In Section 4, we outline the potential contribution of SMMF, together with some concluding remarks, limitations, and suggestions for future research.

2. Literature Review: Multi-Paradigm Practice in Operational Research/Management Science (OR/MS)

During the 1990s, the UK OR/MS and systems community debated the use of more than one methodology (i.e., either through mixing them entirely or simply using parts of them) when tackling complex situations that need some type of intervention. The general term ‘multi-methodology’ was settled to group this practice. In this section, we outline Jackson’s contribution to multi-methodological practice, sketching his work on critical systems and total systems intervention, as well as his systems of systems methodologies (SOSM). We then discuss the ideas of multi-paradigm, multi-methodology, and critical pluralism developed by Mingers. We borrowed his framework for mapping methodologies. Additionally, we use his strategy to frame and design an intervention via the use of the interaction between (what he calls) an ‘Intervention System’ (IS) and the ‘Intellectual Resources’ (IRS). We intend to amalgamate Mingers’ and Jackson’s set of concepts towards designing the proposed systemic multi-methodological framework (SMMF).

2.1. Jackson’s Critical Systems Thinking, Total Systems Intervention, and the Systems of Systems Methodologies

In this section, we sketch the first strand of multi-methodology in OR, which was developed by Jackson. This was mainly through his work on critical systems thinking, total systems intervention, and his systems of systems methodologies.

2.1.1. Critical Systems Thinking (CST)

Jackson’s pluralism [15] (initially named ‘complementarism’ and introduced as part of the total system information approach) identifies the requirements for what he calls a ‘coherent pluralism’ in MS/OR practice as:
  • Flexibility in the use of management science tools. Pluralistic thinking promotes the use of the largest of methods, models, tools, and techniques offered in contemporary management science. Furthermore, methodologies can be ‘decomposed’. Although on this point he agrees with Mingers [16,17], Jackson makes some warnings to prevent relapsing into pragmatism. He also proposes that a methodology should ‘control’ the methods and tools used in the intervention;
  • Paradigm diversity. Pluralistic thinking encourages the application of methodologies serving different paradigms in the same intervention and at all stages;
  • Living with degrees of paradigm incompatibility. Jackson assumes that pluralism must learn to manage degrees of paradigm incompatibility. He denies the possibility of pluralism forming part of a new unique paradigm.

2.1.2. Total Systems Intervention (TSI)

Total systems intervention was created by Flood and Jackson [18] and was originally defined as a novel approach to planning, designing, “problem solving”, and evaluation. TSI, a type of meta-methodology, is underpinned by the main tenets of critical systems thinking (CST), which can be summarised as follows.
(a)
Complementarism is a response to the two prevailing positions in management science, namely (i) pragmatism, which disregards theoretical aspects and focuses on setting up a ‘toolkit’ of techniques that work in ‘practice’, and (ii) isolationism, which closes itself to the use of one method only;
(b)
For social awareness, TSI acknowledges the fact that there are certain methods/methodologies that enjoy more popularity than others;
(c)
For human well-being and emancipation, these elements include Habermas’ ideas about human cognitive concerns. According to Habermas [19], humans have technical (given the importance that humans place on work, which is the first anthropologically based interest); practical (based on the place that humans give to interaction and mutual understanding, this is the other anthropologically based interest); and emancipatory concerns (which is the interest that all humans have in preserving themselves from the constraints imposed by power relations). The latter interest is the one CST claims to support instead of the technical (supported by hard systems approach and cybernetics) or practical (supported by soft systems methods).
It is worth mentioning that, from its beginning, TSI has raised awareness about the issue of human well-being. This is particularly true when the situation involves the use of power relations among stakeholders. As Jackson informs us: “TSI signalled its embrace of the philosophy of CST by declaring its commitment to ‘sociological awareness’, ‘complementarism’, and ‘human well-being and emancipation’” [18]. The project in this study embraced efficiency, effectiveness, and viability. It sought to promote mutual understanding while, at the same time, giving attention to empowerment and emancipation issues. TSI is the operationalisation of Jackson’s multi-methodology approach. It is based on the principle that organisations’ complexity makes it difficult to tackle them with a single model. A system of metaphors and systems methodologies is proposed to achieve a systemic grasp of the situation. TSI comprises three sequential phases, namely creativity, choice, and implementation. This is illustrated below in Table 1.

2.1.3. Systems of Systems Methodologies (SOSM)

In 1984, Jackson and Keys [20] offered a grid of classifying systems methodologies. They were aiming to encourage multi-methodological practice. This is summarised graphically in a two-dimensional framework in what has become the ‘System of Systems Methodologies’ (SOSM). SOSM allows the horizontal axis, the interests and values of the participants involved, to be assessed in three possible scenarios, namely ‘unitary’; ‘pluralistic’; and ‘coercive’. In cases where one group or person exerts power over the rest and no accommodation of values is possible, two grades of complexity exist. These are the ‘simple’ and ‘complex’, which are depicted along the vertical axis.
In 2019, Jackson re-visited the initial grid of problem contexts ‘to make sure it is fitter for purpose’. He made three modifications. He added another element to the vertical dimension (‘complicated’) and changed the name of the horizontal dimension from ‘participants’ to ‘stakeholders’. Finally, the ‘systems’ dimension is turned upside down, as opposed to earlier versions of the grid. Jackson explains that this was conducted to have a more ‘aesthetically pleasing panorama of complexity increasing diagonally from bottom left to top right’ [21] (p. 164).
The revised ‘ideal-type’ grid of system of systems methodologies (SOSM) is depicted in Table 2. It is worth mentioning that the matrix of problem contexts is ideal because they will rarely appear in the real world. For instance, the left-hand side bottom cell (‘Simple-Unitary’) rarely exists in the real world. However, one can argue that a group of soldiers in the army, such as a commando unit, can be given a single objective in a situation that could be described as ‘simple’, in that they need to capture an enemy position (provided that it is a simple case!). Also, it is worth reflecting on the fact that the matrix that Jackson proposes is an epistemological device, and in that respect, it exists in the conceptual world.

2.2. Mingers’ Multi-Paradigm, Multi-Methodology, and Critical Pluralism

Even though Mingers acknowledges the desirability and feasibility of applying multi-methodology in MS/OR interventions, he warns us of the difficulty of trying to put this into practice. Therefore, he proposes two possible ways, namely deploying a framework for mapping methodologies and providing some guidelines about linking parts of the methodologies [16,17].
Mingers argues that, when tackling a problematic situation, the use of a ‘blend of methodologies’ would certainly benefit the agent(s), i.e., the person(s) intervening in the situation or those who are affected by it. He resorts to Habermas’ three worlds (material, social, and personal) to make the following arguments that favour multiple methodologies.
(a)
If a situation is considered inherently complex, such that no single methodology can definitely claim to be able to tackle it, attention should be given to three basic aspects of any intervention. These constitute the realm of the material, the social, and the personal. The understanding of some methodologies might even further clarify some of these three particular aspects;
(b)
An intervention is not a discrete event, but it is actually continuous in nature. It follows that some methodologies would be considered more appropriate for certain stages of the intervention. Therefore, MS/OR practitioners should seriously consider the option of combining different methodologies and partial or entire methods/tools in a customised methodology;
(c)
Pragmatic reasoning favours multi-paradigm multi-methodological practice in MS/OR. In fact, systems practitioners already embrace this approach. Mingers provides us with key examples [16,22].
In the next two sub-sections, we first sketched Mingers’ framework to map methodologies. Then, in preparing to outline our framework, we discuss Mingers’ ideas about the multi-methodological intervention context, in which he examines the relationship between three notional systems. These are (i) the problem content system (PCS), (ii) the intervention system (IS), and (iii) the intellectual resources system (IRS).

2.2.1. Mingers’ Framework to Map Methodologies

Mingers’ study of multi-methodology forms a coherent conceptual and theoretical corpus. It culminates in a framework that allows a set of workable methodologies and methods. His approach to multi-methodology is laid out in [17]. Table 3 below depicts the framework that contains Habermas’ three dimensions of the social, the personal, and the material. The columns in this matrix show the process of intervention by describing the characteristic stages over time. The stages are named appreciation, analysis, assessment, and action. The first stage allows for the problematic system to be surveyed. The second explores the causes, and the third stage proposes ways to control and/or eliminate such causes. The fourth stage concentrates on planning actions.
The simplicity of this grid makes it easy to understand, and also to explain to, researchers and stakeholders. This is especially true when the phases of the systemic intervention (the four ‘As’) are clearly defined (which is often not the case). The features presented in the three dimensions make sense when facing a complex situation. At the same time, its creators, Mingers and Brocklesby [17] still admit the limitations of using Habermas’ theory of knowledge [19]. They warn that this does not guarantee successful navigation across the seas of paradigmatic incommensurability in management science.

2.2.2. Mingers’ Multi-Methodological Intervention Context and the Three Notional Systems: The Problem Content System (PCS), the Intervention System (IS) and the Intellectual Resources System (IRS)

In the late 1980s, Jackson and Keys [23] (p. 198) focused on encouraging multi-methodological practice. They offered a grid of classifying systems methodologies known as the ‘system of systems methodologies’ (SOSM) [24]. Although Mingers’ message (and its potential application) is similar to Jackson’s SOSM, he goes a step further. Mingers actually argues that, in designing the intervention, the two notional systems (from Checkland: i.e., the problem-solving system (PSS) and the problem content system (PCS), are important but certainly not sufficient [16] (p. 419). He argues that this is because the role of the agents is not fully represented in these two systems.
Mingers, therefore, proposes to enrich the intervention context by including the intervention system (IS), which represents the agent(s) and participants engaged in the situation, and the intellectual resources system (IRS). The latter comprises ‘[. . .] frameworks of theories, methodologies, and techniques that can be potentially relevant to the problem situation’ [16] (pp. 419–420). Furthermore, Mingers states that, in order to guide the multi-methodological intervention, we need to reflect on the interplay (labelled as A, B and C) between the three notional systems depicted in Figure 1 [14] (p. 295). In other words, the (i) problem content system (PCS) about the real-world situation; (ii) the intervention system (IS) consisting of the agents engaged; and (iii) the intellectual resources systems (IRS) make up the portfolio of available methodologies and define elements of the team’s expertise.
According to Mingers, each notional system can be regarded as self-referential (indicated by the closed loop attached to each system) to its own past history, in real terms resulting in what it is at a certain point in time. More importantly, the relationships (A, B, and C) between the notional systems represent a key to the intervention context. Mingers suggests that, to have an idea about these relationships, the agent(s) should try to answer a series of questions [14] (p. 298). We have added some of these questions in Figure 1. In summary, Mingers’ proposal essentially resembles Checkland’s soft system methodology (SSM) framework in that, whenever we intervene, we need to be aware of the separation of the two above-mentioned worlds, namely the real world and the systems thinking or conceptual world. The activities depicted in Figure 2 below are useful for designing a systemic intervention, although the order in which the phases were called does not necessarily imply a sequence. Instead, the entire intervention process is enriched with continuous iteration between the ‘Real-world Intervention’ (bottom line) and the ‘reflection about intervention’ (above the line), where the resulting answers to the questions posed by the A, B, and C relationships play a significant role. We believe it is important to have Figure 1 and Table 2 as a backdrop for a systemic intervention. Furthermore, we believe that assessing and reflecting on the outcome of the questions in Figure 1 are crucial for successful systemic intervention because the answers shed light on the intervention. It also guides the way the team should reflect on the intervention design process.
Figure 1 and Figure 2 summarised the core of the concepts of multi-methodology advanced by Mingers and Brocklesby [17] and Mingers [16,22]. In our systemic practice, we normally use the grid in Table 2 as a device to guide the team of stakeholders and researchers on which methods to select. During the systemic intervention, the guidelines in Figure 1 are advised to be followed so that methods can be selected. Subsequently, it is worth reflecting on the choice using Mingers’ multi-methodology process, as shown in Figure 1 and Figure 2. This framework allows for an assessment of the interactions of the three notional systems, which in turn, allows the team to re-design, monitor and re-plan the use of the methods selected. In other words, the interaction of the three notional systems acts as a backdrop and check-out list to inform and further validate the selection of methods.

2.3. Mingers’ Multi-Methodology and Jackson’s Pluralism

In this section, we attempt to summarise and contrast the two more prominent approaches to multi-methodology in MS/OR, namely Jackson’s coherent pluralism, critical systems practice (CSP), and Mingers’ multi-paradigm multi-methodology.
Jackson’s coherent pluralism is the theoretical base for his critical systems practice summarised in his SOSM (Table 2). We summarise below our perspective on the major differences between the two approaches. We also comment on the disagreements between these authors regarding the shortcomings of each one’s approaches. Since we use the main features that are important in the construction of our framework [SMMF], we believe that it is beyond the scope of this article to delve into them in more detail. So, we only present the differences between Jackson and Mingers. Table 4 illustrates the main differences between these two approaches to multi-methodology in management science. As presented later in the next section, the theoretical basis of our proposed framework is underpinned by both Jackson’s and Mingers’ approaches featured in Table 4 below.
The table summarised the main features of the two most recognisable authors working on the themes that discuss multi-methodology in MS. This grid also sketches our view of the earlier differences between Mingers’ and Jackson’s approaches. The table roughly captures their perspectives on multimethodology practice during the late 1990s. Later, in 2005, Mingers [25] expressed his disagreement with Jacksons approach, particularly with the categorisation of Jackson’s ‘problem context’. Apparently, Mingers finds no proper justification for the two dimensions in Jackson’s SOSM [25].
Likewise, Jackson [26] was critical of Mingers’ advocacy of critical realism in management science (MS), labelling his views as one of imperialism in MS, which ultimately prevents the appropriate use of multi-methodology. “Committing oneself to critical realism, as does Mingers, prevents you from also being multimethodological. Mingers says that critical realism is happy ‘to accept the validity of a wide range of research methods’, but this is true only so long as they can be used to its own ends. […] This is not supporting pluralism or multimethodology in management science but advocating an ‘imperialist’ strategy.” [26] (p. 1370).
It is not our intention to revive this debate. We believe that the debate in the 2010s has continued with new insights about how to handle multi-methodological practice. For the purposes of this paper, we want to use the features depicted in Table 3 and Figure 2 to support the building of our framework. It is well-acknowledged that Jackson and Mingers have contributed to strengthening the field of multi-methodological practice in MS. We would like to rescue their key concepts. It is also important to stress that the interpretation of the features of Jackson’s coherent pluralism and Mingers’ multi-paradigm multi-methodology, represented in Table 4, is purely ours. Arguably, as researchers and readers, we should be able to produce frameworks based on our own way of reading and understanding those ideas. We believe that making a point about who is correct and who is not is irrelevant when practitioners and researchers want to apply those ideas in the real world and learn from them.

3. A Multi-Methodological Framework (SMMF) to Explore and Reflect a Systemic Intervention

As stated earlier, Mingers’ and Jackson’s work has definitely helped to advance multi-methodological practice. Both have produced frameworks to guide multi-methodological practice from their own theoretical bases. Mingers’ framework for mapping methodologies [16,17] and Jackson’s systems of systemic methodologies [21,27] have been widely used with various degrees of success amongst the OR and systems communities.
Figure 3 depicts our attempt to amalgamate Jackson’s and Mingers’ concepts and ideas about how to guide and design a multi-methodological and multi-paradigm intervention. The shape of the proposed framework is a theatre layout. The three-dimensional picture presents various levels or areas that form a traditional ‘theatre’, in this case, the one in which the systemic intervention is going to be ‘enacted’ and operated in practice.
We use the imagery of a theatre to depict the proposed systemic framework. Generally, theatres are divided into two main parts or areas, with the stage and the audience facing and mirroring each other and separated by the main curtains (or ‘apron’). The curtains play the practical function of separating the actors and the audience, but they also work as a dividing device between two worlds, namely the real and the fictional. In terms of systemic thinking and following the seminal distinction made by Checkland, this line divides the real world from the systemic world [28,29].
In Figure 3, we have two columns. In this case, the first outlines Jackson’s taxonomy of ‘Systems’, namely ‘complex’; ‘complicated’; and ‘simple’. The other column completing the backdrop represents Habermas’ worlds, namely ‘Social’, ‘Personal’, and ‘Material’. We believe that, in practice (that is when we face a real situation), synergies between the elements represented in the columns can be highlighted. In human affairs and social interventions, Habermas’ ‘material’ world could be aligned with Jackson’s ‘simple’ categorisation of the system, and the ‘personal’ world may suggest dealings with a ‘complicated’ system. Finally, when we operate in the domain of the ‘social’ world, we expect to face more ‘complex’ systems.
The proposed framework resembles the theatrical stage in that it has two ‘walls’: On the left-hand side of the stage, we see Jackson’s SOSM (with some suggested systems methodologies). The right-hand side of the stage exhibits the well-known Mingers’ multi-methodological framework/map in which a combination of methodologies/methods are selected to tackle the particular issues arising when intervening within the context of the Habermas’ worlds (Social, Personal and Material). For example, a combination (marked with ‘*’ in Figure 3) could be as follows: Soft Systems Methodology (SSM) is selected to be suitable for both the Appreciation and the Analysis stage; the Viable Systems Model (VSM) as suitable for the Assessment stage; and other methodology from the team’s intellectual resource system (IRS) could be selected as suitable for the Action stage. It is the interaction between the methodologies contained on the ‘bricks’ of these two walls that will render a suitable, effective combination of the methodologies selected to address the problematic situation.
In what will be the stage of the theatre, we have located the three notional subsystems alluded to by Mingers, namely the problem content system (PCS); the intellectual resources system (IRS); and the intervention systems (IS). As explained above, the interactions of these systems (A, B, and C) are important in order to assess and better guide the intervention. Two of the three interactions are, to some extent, dealt with by the SSM (“B”, the relationship between PSC and IS, is SSM Analysis 1; and “C”, the relationship between IRS and PSC, is SSM Analyses 2 and 3). Therefore, in the application of the proposed framework, we will concentrate on “A” (the relationship between IS and IRS). Details about how this interaction was built in the framework are provided at the end of this section.

3.1. Intellectual Resources System (IRS): Exploring the Project Team Skills and Abilities

To illustrate the consequences of taking stock of IRS in an intervention, we describe how this was conducted in a previous project in which a team of UK and Argentinean researchers participated [9]. The team responsible for the intervention included engineers with a leaning towards production. Beyond that common denominator, the participants had different formations and competencies. All have training and experience in methodologies of problem structuring. In particular, the coordinator had a strong background in process management and the development of related systems.
The second team member had excellent training and experience in soft operations research, with an emphasis on soft system methodology. In addition, this person had significant competency with multi-methodological interventions. The third member complemented his academic activity as a consultant with different organizations. Notably, he had distinctive competencies in the design and development of organizational learning processes.
As stated above, Mingers advises that, after the initial plan of the methodologies is deployed, the project team and the stakeholders need to reflect and constantly review the particular design. In order to achieve this, it is vital to focus on the relationships that span the three notional systems depicted in Figure 1. In this section, we discuss the way that this advice can be activated and put to work. We concentrate on the way we used the intellectual resource system (IRS) as a means of exploring the project team’s skills and abilities. Mingers, as far as we can make it, does not suggest any way of exploring the features of the IRS in terms of the personal characteristics of each member of the research team. As an illustration of how this can be conducted in practice, we propose exploring the personalities of the IRS team and suggest that this can be accomplished by using a simple procedure for determining the personality types of the key team members. It is worth mentioning that there may be other ways of exploring or testing the research team’s personality types.
The way we understand the use of the IRS is that the research team has to reflect on the questions arising when the IRS interacts with the ‘Intervention System (IS)‘ which includes agent(s) undertaking an intervention. As we can see in Figure 1, when IRS and IS interact, Mingers poses the following questions that need to be addressed:
  • what is the level of critical awareness/understanding of potential methodologies?
  • what is my experience and skill in using them?
  • what is my personality/cognitive style comfortable with?
  • to what extent can I work in varied paradigms?
  • etc.
In our experience, questions (1) and (2) are fairly easy to address/complete, and it is something that the research team has conducted in the process of the intervention. In general, we answered questions (1) and (2) by reflecting (through conversations) on the potential methodologies from a critical perspective and by assembling an inventory of the research team skills/capabilities that are, to some extent, known or declared before or during the intervention. That said, it is clear that, if after reflecting on the answer to these questions, the ‘inventory of skills’ produces gaps in the team’s abilities, then the team should address them through training.
On the other hand, questions (3) and (4) are more difficult to address, and as we said previously in this article, Mingers and other authors do not give much guidance on how to do so. From our experience working with teams, one of the main difficulties is to break down the psychological barriers between team members, and one way of doing that will be to attempt to share/compare the types of cognitive styles and/or personality types. MBTI could be a first step to starting a conversation about cognitive styles and personality types that may or may not break those psychological barriers, but it is worth a try. In the case of working on the Anglo-Argentinean project, it was a first step. It helped a little, as it signalled some similarities/differences in the way researchers approached a situation.
It is worth mentioning that skills learned and capabilities acquired are, in some way, adopted externally and are always evolving. Personality types are, in some ways, more permanent and, in some ways, more difficult to deal with/manage within the dynamics of teamwork. Therefore, having a grasp of the personality types/cognitive styles can help the group with possible ways of harnessing these different personality types. Below is an illustration from our previous experience.
During the project, the Anglo-Argentinean team decided to use the psychological personality types developed by Carl Jung [30], the founder of analytical psychology. Jung’s theory of psychological types is certainly one of the most influential approaches to personality typology. Early in the 20th century, Katharine Cook-Briggs was interested in personality types. Together with her daughter, Isabel Briggs-Myers, she developed a way to describe Jung’s different personality categories [31]. According to Jung, we perceive/become aware of something in two different ways, namely by (i) sensing (hence, by means of conscious use of our sense, we become aware of objects and people) and through (ii) intuition, which is when we perceive via ideas or associations. These ideas or associations are processed at an unconscious level beyond our perception.
Jung states that we judge events or objects (that is, we conclude about something) in two contrasting ways. These are when (i) we judge by a logical, quite impersonal objective approach, or by thinking, and (ii) when we judge by feeling, or by a subjective approach. This latter practice attaches subjective value to our perceptions and is more feeling-based.
These characteristics are expressed in the form of two types of personality, namely one dominating and another subordinating. According to Jung, these are accommodated in the following way. If one personality is perceptual, the other (subordinating) is judgmental and vice versa. In other words, one person combines (two personalities) thinking with intuition, while another may combine feeling with sensation. Whatever the case, there is no room for combining thinking with feeling or sensation with intuition because they are mutually exclusive.
Jung also argues that the way or style in which we make judgments is also important. For example, a person can be (i) controlled, structured, or judgmental or (ii) adaptive, spontaneous, and perceptual. Jung also distinguishes between the well-known propensity of a person to be (i) introverted (or inner-directed) or (ii) to be extroverted (or outer-directed). This typology has been used by Myers [31] to create an inventory (the so-called Myers–Briggs Type Indicator or simply MBTI) which essentially indicates 16 discrete types of personality generated from a combination of the following characteristics:
  • two ways of perceiving: sensing (S) or intuition (N);
  • two ways of forming conclusions: thinking (T) or feeling (F);
  • two sources and focus of energy: introverted (I) or extroverted (E);
  • two styles/orientations: judgmental (J) or perceptual (P).
With a total of 2 × 2 × 2 × 2, in reality, 16 outcomes are possible. The first personality type is STIJ. That is, sensing, thinking, introverted and judgmental. The next would be STIP. That is, sensing, thinking, intuiting, and perceptual, and so on.
In order to find out the personality profile of the team members in this study, we used the “Free Personality Test” organised by NERIS Type Explorer® (https://www.16personalities.com/free-personality-test, accessed 10 October 2021). NERIS has another variable, namely identity, which allows for an assessment of how confident individuals are in their abilities and decisions https://www.16personalities.com/articles/our-theory (accessed 10 October 2021). We decided to stick to the initial four dimensions developed by Myers–Briggs and based on Jung’s initial idea, summarised in the Myers–Briggs Type Indicator® (MBTI®) [31]. Table 5 resumes the results of the Free Personality Test carried out by the Anglo-Argentinean team [9].
From Table 5, we can deduce that the project team exhibits a good balance of personality traits and has a relevant set of cognitive skills to carry out the intervention. Overall, these characteristics ensure that the members are open to using tools that involve both hard and soft features. Two team members are introverted types. This can be an advantage (reflecting type) but also a disadvantage, since a moderate extroverted attitude is needed to facilitate an intervention (particularly, in this case with the SME owner/managers and/or staff). Considering the fact that three members have a first degree in engineering and mathematics, it was encouraging to see that the personality profiles indicate that they are also open to less “analytic” or “logical” tasks.
As it can be seen, only team member ‘A’ exhibits “thinking”, with the percentage slightly over 50. The other two members are more “feeling” types. Generally, this could be seen as a tendency to explore the softer issues of a problematic situation. The analysis of the last personality trait displayed by the three members, “judgment” (with a percentage of at least 57 for all three members) seems to indicate that the team has the skills to make judgments in a controlled and structured manner.
It is worth mentioning that there may be other ways of exploring or testing the research team’s personality types. Another one could be the well-known Nine Belbin Team Roles (https://www.belbin.com/, accessed 15 November 2024). Competencies in Belbin’s view are skills and an understanding of methods and methodologies, which are effectively the ‘task focus’. That is the core of skills to tackle the job. If we relate these ideas to the systems intervention strategy reported in this article, Belbin’s task focus corresponds to the likely answers to questions (1) and (2) above. The second vital element in Belbin’s model is the ‘team focus’, which involves the dynamic process of collaboration within a group and is crucial to improving the team’s efficacy and efficiency. These can be loosely related to questions (3) and (4) above and can be used in conjunction to explore possible relationships between personality types and team roles. That said, there is no space in this article to delve into this topic in-depth but to enquire about these roles within the team can clearly be used to enhance the team performance. Furthermore, in our opinion, Belbin’s roles will be a sort of product/externalisation of a particular combination of the MBTI personality types.

3.2. The ‘Theatre’ of a Systemic Intervention Depicting the Multi-Methodological

In continuing with the description of the framework depicted in Figure 3, evidently, with the three notional systems at the centre (and straddling between the stage and the audience), we have a problematic situation in need of improvement. The one on the left-hand side contains a representation of the agents who are crucial in designing and driving the intervention. This involves the project team and organisation participants (including the problem owner and problem solver). This element is also crossing the two worlds. Thus, indicating that the agents act in the sense that they have to use concepts to bring real action in the real world. The element on the right indicates that we need to pay attention to the other stakeholders with a declared interest in the situation (as indicated by the interaction B and C of the three notional systems). The lines forming circular arrows in both elements, “Agents” and “Other stakeholders”, are meant to indicate this continuous traffic of the people involved in the intervention and travelling between the two worlds.
When designing an intervention, the actual process is a critical part of the strategy. This includes the selection of the methodologies to be combined with the identification of the notional systems that guide the intervention. The intervention team needs to trigger the interaction between the notional systems and try to assemble answers to the questions posed (see boxes A, B, and C in Figure 1). Answering these questions will be a starting point to conceptualise the real-world situation to what we believe constitutes the problem. This is enacted in both worlds, the systemic and the real. These factors in planning the strategy of the intervention are discussed below in more detail. We assemble them in a grid that reflects what the systemic framework proposes by amalgamating both Mingers’ and Jackson’s approaches.

4. Potential Uses of SMMF, Our Research Contributions, and Research Limitations

In the following paragraphs, we report on both the design and the reflection arising from implementing the set of methodologies. As we said in the introduction, an initial draft of the framework was applied to an Argentine textile SME. In recounting the intervention, we adhere to the four As scheme. We start the recounting with our reflection on the consequences (for the project team) of the interaction involving the three notional systems. The reader should note that, in a strict sense, this separation (reflection and design) is very theoretical and difficult to dissect. As it turned out, in practice, the flow between design and reflection occurred many times and was adjusted as the intervention progressed. Also, when reporting the use of the methodologies, we concentrate on the most relevant. However, we are unable to report the use of all of them due to this article’s word limit.
The driving RQ of this paper was to propose a theoretical argument for amalgamating (in a single conceptual framework), Jackson’s set of concepts of critical systems thinking and Mingers’ viewpoints on methodology selection and Mingers’ ideas on the importance of using three notional systems (the problem content system, the intellectual resources system, and the intervention system), when designing a systemic intervention. We believe that the framework we offer here is a possible amalgamation and that, as such, it can contribute to enhancing systems interventions. Below, we highlight the main contributions of this article.

4.1. Potential Theoretical Contributions

The theoretical contribution offered here can be seen as a timely addition to the current OR/MS research and practice, in which a number of systemic methodologies are utilised under one framework. We proposed to develop this further by building a case for multi-methodological integration to improve a problematic situation. This was conducted by designing a framework in which we incorporated ideas and concepts of (a) Mingers’ and Brocklesby’s four-phase grid to guide an intervention, (b) Mingers’ three notional systems (the problem content system, the intellectual resources system, and the intervention system), and (c) Jackson’s system of systems methodologies.
Mingers [22] argues that, when creating an intervention using a multi-methodological approach, a number of barriers could prevent the feasibility of multi-method research. He advises the researcher to be mindful of what he calls philosophical, cultural, and psychological feasibility [22] (p. 13). In this paper, while planning our intervention, we were aware of two of these barriers (cultural and psychological). We tried to overcome them by focusing on designing an initial intervention (selecting a portfolio of methodologies) and, then, reflecting on and adjusting the design. This was conducted by analysing the selection of our thoughts through the lens of the above-mentioned three notional systems.
With regards to the cultural feasibility of an intervention, we can offer here our experience when an Anglo-Argentinean research team worked on an application of an earlier version [9,10] of the framework here presented. Throughout the discussion of all stages of the collaboration project, we were very aware of these potential cultural difficulties. However, this was minimised because the team members had already gained experience in both hard and soft OR interventions. They all also shared a Latin American cultural background. This helped with dialogue and improved openness among team members. Having these insights beforehand precluded any further difficulties linked to potential sub-cultures and privileging one method over another. Furthermore, we also overcame the cognitive barriers (psychological feasibility). We gained insight into how to solve these barriers by assembling a psychological profile of the team members. This was used to identify any common psychological features and to discuss and accommodate personality differences.

4.2. Final Remarks and Limitations

One important point of clarification about the two approaches used is that we used Mingers’ and Jackson’s views on how to approach multi-methodological practice in MS. We are aware that the amalgamation we proposed may be controversial and not seen as feasible by these authors and/or their followers. In our view, both multi-methodological approaches are useful and not mutually exclusive (we contest that, in multi-methodological practice, nothing is exclusive). In fact, Mingers actually asserts that travelling along paradigms is certainly possible. Mingers’ simple grid and his sophisticated explanation and Jackson’s SOSM are both epistemological devices (mental models). It should be left to researchers, as the authors of the paper, to judge for themselves if the amalgamation of the two sets of concepts makes sense and proves to be useful or not.
We have to admit that the framework we suggested is based mainly on the ideas of Jackson and Mingers. That said, we acknowledge that Midgley [32] has also contributed substantially to enhancing a systemic intervention through his ‘systemic intervention approach’, Midgley [32] (pp. 128–133). In particular, his proposed ‘Facilitated boundary critique’ supported stakeholders in the exploration of complex problems. Midgley’s aspects of a methodology for systemic intervention (boundary judgments; making choices concerning theory and methods; and taking action for improvement) are particularly relevant to further refining the framework here proposed. That said, for now, we acknowledge that our framework has the limitation of not incorporating Midgley’s systemic intervention concepts, but we will consider this in future research.
Finally, we are aware that what we offer here is just half the story, as there is no application of the framework proposed here. The main reason is that all frameworks are guidelines, and it is a prerogative of the practitioners to adopt/adapt to their own circumstances. Although an initial draft of the framework was applied to an Argentine textile SME, we believe that an application merits a full paper on its own. We hope to apply fully this new framework in the future.

Author Contributions

Conceptualization, A.P.-C.; methodology, A.P.-C.; software, L.A.; validation, A.P.-C.; formal analysis, A.P.-C. and L.A.; investigation, A.P.-C.; resources, A.P.-C.; data curation, L.A.; writing—original draft preparation, A.P.-C.; writing—review and editing, A.P.-C. and L.A.; visualization, L.A.; supervision, A.P.-C.; project administration, A.P.-C. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

No new data was created.

Acknowledgments

The authors would like to thank Toni Burrowes-Cromwell for her comments and insights, which greatly improved the first version of this paper. The authors also thank the anonymous reviewers for their valuable comments and suggestions.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Franco, L.A.; Cushman, M.; Rosenhead, J. Project review and learning in the construction industry: Embedding a problem structuring method within a partnership context. Eur. J. Oper. Res. 2004, 152, 586–601. [Google Scholar] [CrossRef]
  2. Hindle, G.A.; Franco, L.A. Combining problem structuring methods to conduct applied research: A mixed methods approach to studying fitness-to-drive in the UK. J. Oper. Res. Soc. 2009, 60, 1637–1648. [Google Scholar] [CrossRef]
  3. Pollack, J. Multimethodology in series and parallel: Strategic planning using hard and soft OR. J. Oper. Res. Soc. 2009, 60, 156–167. [Google Scholar] [CrossRef]
  4. Paucar-Caceres, A.; Hart, D.; Vergés, J.R.; Sierra-Lozano, D. Applying Soft Systems Methodology to the Practice of Managing Family Businesses in Catalonia. Syst. Res. Behav. Sci. 2016, 33, 312–323. [Google Scholar] [CrossRef]
  5. Paucar-Caceres, A.; Jerardino-Wiesenborn, B. A bridge for two views: Checkland’s soft systems methodology and Maturana’s ontology of the observer. J. Oper. Res. Soc. 2020, 71, 660–672. [Google Scholar] [CrossRef]
  6. Paucar-Caceres, A.; Rodriguez-Ulloa, R. An application of Soft Systems Dynamics Methodology (SSDM). J. Oper. Res. Soc. 2007, 58, 701–713. [Google Scholar] [CrossRef]
  7. Henao, F.; Franco, L.A. Unpacking multimethodology: Impacts of a community development intervention. Eur. J. Oper. Res. 2016, 253, 681–696. [Google Scholar] [CrossRef]
  8. Abuabara, L.; Paucar-Caceres, A.; Belderrain, M.C.; Burrowes-Cromwell, T. A systemic framework based on Soft OR approaches to support teamwork strategy: An aviation manufacturer Brazilian company case. J. Oper. Res. Soc. 2018, 69, 220–234. [Google Scholar] [CrossRef]
  9. Castellini, M.A.; Paucar-Caceres, A. A Conceptual Framework for Integrating Methodologies in Management: Partial Results of a Systemic Intervention in a Textile SME in Argentina. Syst. Res. Behav. Sci. 2019, 36, 20–35. [Google Scholar] [CrossRef]
  10. Castellini, M.A.; Paucar-Cáceres, A. Marco multimetodológico para el diseño e implementación de una intervención sistémica. Rev. De La Esc. De Perfecc. En Investig. Oper. 2022, 30, 5–22. Available online: http://id.caicyt.gov.ar/ark:/s18539777/68q38gxeg (accessed on 26 September 2023).
  11. Duran-Encalada, J.A.; Paucar-Caceres, A. System Dynamics Urban Sustainability Model for Puerto Aura in Puebla, Mexico. Syst. Pract. Action Res. 2009, 22, 77–99. [Google Scholar] [CrossRef]
  12. Munro, I.; Mingers, J. The use of multimethodology in practice—Results of a survey of practitioners. J. Oper. Res. Soc. 2002, 53, 369–378. [Google Scholar] [CrossRef]
  13. Paucar-Caceres, A.; Ribeiro dos Santos, P.; Wright, G.; Belderrain, M.C.N. Soft situational strategic planning (SSSP): A method and case study of its application in a Brazilian municipality. J. Oper. Res. Soc. 2019, 71, 363–380. [Google Scholar] [CrossRef]
  14. Mingers, J. Multi-Methodology-Mixing and Matching Methods. In Rational Analysis for a Problematic World Revisited, 2nd ed.; Rosenhead, J., Mingers, J., Eds.; John Wiley & Sons: Chichester, UK, 2001. [Google Scholar]
  15. Jackson, M.C. Towards coherent pluralism in management science. J. Oper. Res. Soc. 1999, 50, 12–22. [Google Scholar] [CrossRef]
  16. Mingers, J. Towards critical pluralism. In Multi-methodology: Towards Theory and Practice and Mixing and Matching Methodologies; Mingers, J., Gill, A., Eds.; Wiley: Chichester, UK, 1997; pp. 407–440. [Google Scholar]
  17. Mingers, J.; Brocklesby, J. Multimethodology: Towards a framework for mixing methodologies. Omega 1997, 25, 489–509. [Google Scholar] [CrossRef]
  18. Flood, R.L.; Jackson, M.C. Total systems intervention: A practical face to critical systems thinking. Syst. Pract. 1991, 4, 197–213. [Google Scholar] [CrossRef]
  19. Habermas, J. Knowledge and Interest. In Sociological Theory and Philosophical Analysis; Emmet, D., MacIntyre, A., Eds.; Palgrave Macmillan: London, UK, 1970; pp. 36–54. [Google Scholar] [CrossRef]
  20. Jackson, M.C.; Keys, P. Towards a System of Systems Methodologies. J. Oper. Res. Soc. 1984, 35, 473–486. [Google Scholar] [CrossRef]
  21. Jackson, M.C. Critical Systems Thinking and the Management of Complexity; Wiley: Chichester, UK, 2019. [Google Scholar]
  22. Mingers, J. Multi-paradigm multimethodology. In Multimethodology: The Theory and Practice of Combining Management Science Methodologies; Mingers, J., Gill, A., Eds.; John Wiley & Sons: Chichester, UK, 1997; pp. 1–20. [Google Scholar]
  23. Jackson, M.C.; Keys, P. New Directions in Management Science; Gower Pub Co: Swansea, UK, 1987. [Google Scholar]
  24. Jackson, M.C. Systems Methodology for the Management Sciences, 1st ed; Springer Science & Business Media: Berlin/Heidelberg, Germany, 1991. [Google Scholar]
  25. Mingers, J. ‘More dangerous than an unanswered question is an unquestioned answer’: A contribution to the Ulrich debate. J. Oper. Res. Soc. 2005, 56, 468–474. [Google Scholar] [CrossRef]
  26. Jackson, M.C. John Mingers is a critical realist imperialist. J. Oper. Res. Soc. 2006, 57, 1370–1371. [Google Scholar] [CrossRef]
  27. Jackson, M.C. Systems Thinking: Creative Holism for Managers; John Wiley & Sons: Chichester, UK, 2003. [Google Scholar]
  28. Checkland, P. Systems Thinking, Systems Practice; John Wiley & Sons: Chichester, UK, 1981. [Google Scholar]
  29. Checkland, P. Systems Thinking, Systems Practice: Includes a 30-Year Retrospective; John Wiley & Sons: Chichester, UK, 1999. [Google Scholar]
  30. Jung, C.G. Psychological Types; Routledge: London, UK, 1991. [Google Scholar]
  31. Myers, I.B. The Myers-Briggs Type Indicator: Manual (1962); Consulting Psychologists Press: Palo Alto, CA, USA, 1962. [Google Scholar] [CrossRef]
  32. Midley, G. Systemic Intervention—Philosophy, Methodology and Practice (Contemporary Systems Thinking); Springer Science & Business Media: New York, NY, USA, 2000. [Google Scholar]
Figure 1. Context of a multi-methodology intervention (from [14] (p. 295)).
Figure 1. Context of a multi-methodology intervention (from [14] (p. 295)).
Systems 12 00527 g001
Figure 2. Designing a systemic intervention (from [14] (p. 301)).
Figure 2. Designing a systemic intervention (from [14] (p. 301)).
Systems 12 00527 g002
Figure 3. The ‘Theatre’ of a systemic intervention depicting the multi-methodological. Context: Jackson’s and Mingers’ notional systems and the real-world system of an intervention. Source: the authors.
Figure 3. The ‘Theatre’ of a systemic intervention depicting the multi-methodological. Context: Jackson’s and Mingers’ notional systems and the real-world system of an intervention. Source: the authors.
Systems 12 00527 g003
Table 1. The TSI meta-methodology (from [18] (p. 54)).
Table 1. The TSI meta-methodology (from [18] (p. 54)).
Creativity
TaskTo highlight significant concerns, issues, and problems.
ToolsCreativity-enhancing devices, especially systems metaphors.
OutcomeDominant and dependent concerns, issues, and problems identified.
Choice
TaskTo choose an appropriate systems intervention methodology or methodologies.
ToolsMethods for revealing the strengths and weakness of different systems methodologies (e.g., the SOSM).
OutcomeDominant and dependent methodologies chosen for use.
Implementation
TaskTo arrive at and implement specific positive change proposals.
ToolsSystems methodologies employed according to the logic of TSI.
OutcomeHighly relevant and coordinated change that secures significant improvement in the problem situation.
Table 2. A revised ‘ideal-type’ grid of problem contexts (from [21] (p. 164)).
Table 2. A revised ‘ideal-type’ grid of problem contexts (from [21] (p. 164)).
STAKEHOLDERS
UNITARYPLURALISTCOERCIVE
SYSTEMSCOMPLEXComplex-UnitaryComplex-PluralistComplex-Coercive
COMPLICATEDComplicated-UnitaryComplicated-PluralistComplicated-Coercive
SIMPLESimple-UnitarySimple-PluralistSimple-Coercive
Table 3. A framework for mapping methodologies (adapted from [17]).
Table 3. A framework for mapping methodologies (adapted from [17]).
Dimension
of the Problem
(from Habermas)
The 4 ‘As’ Phases
AppreciationAnalysisAssessmentAction
SocialSocial practices, power relationsDistortions, conflicts, interestsWays of altering existing structuresGenerate empowerment and enlightenment
PersonalIndividual beliefs, meanings, emotionsDiffering perceptions and personal rationalityAlternative conceptualization and constructionsGenerate accommodation and consensus
MaterialPhysical circumstancesUnderlying causal structureAlternative physical and structural arrangementsSelect and implement best alternatives
Table 4. Jackson’s coherent pluralism: critical systems practice (CSP) and Mingers’ multiparadigm.
Table 4. Jackson’s coherent pluralism: critical systems practice (CSP) and Mingers’ multiparadigm.
Coherent Pluralism: Critical Systems PracticeMulti-Paradigm Multi-Methodology: Critical Pluralism
  • Combine tools; methods; models; and techniques but not parts of methodologies;
  • Need for a meta-methodology to manage paradigm diversity (‘conversation’ between them) and relationship between methodologies;
  • Accept to live with degrees of paradigmatic incompatibility;
  • Availability of theoretically informed range of methodologies is crucial;
  • Critical systems practice is the meta-methodology needed to operationalise coherent pluralism.
  • Combine methodologies; methodological stages; techniques; and tools;
  • Multi-paradigmatic intervention is feasible. Methodologies from different paradigms work together in the intervention; there is no need for a meta-methodology;
  • It is difficult but possible to ‘switch’ between paradigm subcultures: philosophical feasibility;
  • Need to move away from abstract theory to the practical needs (worlds) of the agent;
  • Multi-paradigm multi-methodology is supported by critical pluralism.
Table 5. Individual traits of the Anglo-Argentinean team members.
Table 5. Individual traits of the Anglo-Argentinean team members.
Team MembersIndividual Traits
A (ESTJ)Extroverted (58%)Sensing (59%)Thinking (58%)Judging (58%)
B (INFJ)Introverted (51%)Intuitive (56%)Feeling (54%)Judging (61%)
C (ISFJ)Introverted (83%)Sensing (60%)Feeling (67%)Judging (57%)
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Paucar-Cáceres, A.; Abuabara, L. A Multi-Methodological Conceptual Framework to Explore Systemic Interventions. Systems 2024, 12, 527. https://doi.org/10.3390/systems12120527

AMA Style

Paucar-Cáceres A, Abuabara L. A Multi-Methodological Conceptual Framework to Explore Systemic Interventions. Systems. 2024; 12(12):527. https://doi.org/10.3390/systems12120527

Chicago/Turabian Style

Paucar-Cáceres, Alberto, and Leila Abuabara. 2024. "A Multi-Methodological Conceptual Framework to Explore Systemic Interventions" Systems 12, no. 12: 527. https://doi.org/10.3390/systems12120527

APA Style

Paucar-Cáceres, A., & Abuabara, L. (2024). A Multi-Methodological Conceptual Framework to Explore Systemic Interventions. Systems, 12(12), 527. https://doi.org/10.3390/systems12120527

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop