Establishing the Foundations to Measure Organizational Agility for Military Organizations

: There is an ongoing demand for organizations to become more agile in order to prosper amongst their competitors. Many military organizations have declared a renewed focus towards organizational agility. The goal of this research is to isolate the variables needed to measure organizational agility (OA) in military organizations, allowing for the future development of a suitable method to measure OA without the need to interact with outside organizations. This article begins by providing a suitable and formal deﬁnition of organizational agility by exploring and analyzing relevant scholarly literature on the subject. Related terms, such as organizational resiliency, ﬂexibility, robustness, versatility, and adaptability are also explored to examine their deﬁnition boundaries and any overlapping areas. Existing methods to measure organizational agility are examined and summarized, and the current limitations to their application are highlighted. Previous studies to ﬁnd characteristics associated with organizational agility were also examined, and an initial set of 88 organizational agility characteristics was built. Since these included possible redundant or overlapping characteristics, the Q-sort method was employed to discover, analyze, and eliminate redundant items from the dataset, ultimately resulting in 64 unique characteristics. The result is a suitable deﬁnition for organization agility applicable to military organizations and a list of potential associated characteristics that summarizes related research to date. This groundwork establishes the foundation to conduct a multi-organization study to further reﬁne the characteristic list and ultimately develop a method to measure organizational agility. Versatility: the measure of how broadly [an organization’s] capability extend in terms of foreseeable and unforeseeable sources of change .” the measure of ﬀ ectively [an organization] can modify its own capabilities in response to change after it has been ﬁelded


Introduction
Over the last decade we have seen smaller, more efficient agile organizations outmaneuver traditionally established institutions. The pace of change has accelerated throughout the information age; an age where information is readily available, and transformative technologies can topple legacy designs overnight. Although particularly evident in the business sector, this phenomenon has also gained significant momentum in the defense sector. The President, Department of Defense (DoD) executives, Congress, and the service chiefs have all come to the same conclusion; that a more agile, flexible, and technologically advanced force is needed to outdo their adversaries [1].
Nation-state militaries spend a significant amount of financial resources and are expected to succeed against their opponent, yet oftentimes, they do not directly engage with their opponents for decades at a time. What happens in a sector where innovation and agility are both vitally important, but a timely and consistent feedback mechanism to measure one's progress is virtually non-existent? Although the true test of a military is during a turbulent period of engagement with an opponent, interim methods must be developed to measure each critical organizational trait. Table 1. Summary of organizational agility definitions .
Year Author(s) Definition Capability Capacity Goldman, Nagel & Preiss (L. Goldman et al., 1995) Firms ability to cope with rapid, relentless, and uncertain changes and thrive in a competitive environment of continually and unpredictably changing opportunities. X X 1995 Gehani (Gehani, 1995) An agile organization can quickly satisfy customer orders; can introduce new products frequently in a timely manner; and can even get in and out of its strategic alliances speedily. X 1996 Cho, Jung, Kim (Cho et al., 1996) Capability of surviving and prospering in a competitive environment of continuous and unpredictable change by reacting quickly and electively to changing markets, driven by customer-designed products and services X 1997 Morgan (Morgan, 1997) Internal operations at a level of fluidity and flexibility that matches the degree of turmoil in external environments. X

1998
Dyer & Shafer (Dyer & Shafer, 1998) Capacity to be infinitely adaptable without having to change . . . necessary core competence for organizations operating in dynamic external environments . . . develop a built-in capacity to shift, flex, and adjust either alone or with alliance partners, as circumstances change.
X X 1998 Kidd (Kidd, 1995) Unites organizational processes and people with advanced technology to meet customer demands for customized high quality products and services in a relatively short timeframe. X 1998 Feng and Zhang, 1998) An agile enterprise could swiftly reconfigure operations, processes, and business relationships, thriving in an environment of continuous and unpredictable change. X 1999 Sharifi and Zhang (1999) The ability to cope with unexpected changes, to survive unprecedented threats of business environment, and to take advantage of changes as opportunities. X 1999 Yusuf, Sarhadi, Gunasekaran (Yusuf et al., 1999) Agility is the successful exploration of competitive bases through the integration of reconfigurable resources and best practices in a knowledge-rich environment to provide customer-driven products and services in a fast changing market environment.

X 2001
Dove (Dove, 2002) Providing the potential for an organization to thrive in a continuously changing, unpredictable business environment. X

2001
Grewal & Tansuhaj (Grewal & Tansuhaj, 2001) Organizational ability to manage economic and political risks by promptly responding in a proactive or reactive manner to market threats and opportunities. X

2003
Alberts & Hayes (Albert & Hayes, 2003) The synergistic combination of robustness, resilience, responsiveness, flexibility, innovation, and adaption. X X 2006 Van Oosterhout, et al., 2006) The ability to swiftly and easily change businesses and business processes beyond the normal level of flexibility to effectively manage unpredictable external and internal changes. X 2008 Erande, Verma (Erande & Verma, 2008) Ability to respond to unpredictable changes with quick response and profitability. X

2008
Doz & Kosonen (Kosonen & Doz, 2007) Capacity to continuously adjust and adapt strategic direction in a core business to create value for a company. X 2009 Worley & Lawler (Worley & Iii, 2009) Dynamic organization design capability that can sense the need for change from both internal and external sources, carry out those changes routinely, and sustain above average performance. X X Capacity of an organization to efficiently and effectively redeploy/redirect its resources to value creating and value protecting (and capturing) higher-yield activities as internal and external circumstances warrant X 2020 Walter (Walter 2020) Organizational Agility is a learned, permanently-available dynamic capability that can be performed to a necessary degree in a quick and efficient fashion, and whenever needed in order to increase business performance in a volatile market environment. It was found that many authors blur the line between capability and capacity and, far too often, mistakenly use them interchangeably. Capacity is an ability that exists at present and capability represents a higher level ability that can be achieved in the future [25]. Each definition in Table 1 was evaluated to determine the intended context and assessed whether it represented a capacity, capability, or both. Of the 24 definitions of organizational agility, 10 were categorized as capacity; 10 as a future capability; and four provided a mix of capacity and capability. A breakdown of the important components was achieved by analyzing the specific words and meaning within these definitions. As shown in Figure 1, the most repeated components of the definition are "rapid response" and "stimuli is external environment." These are followed closely by "customer driven output," "environment of uncertainty," and "opportunistic outcome." It was found that many authors blur the line between capability and capacity and, far too often, mistakenly use them interchangeably. Capacity is an ability that exists at present and capability represents a higher level ability that can be achieved in the future [25]. Each definition in Table 1 was evaluated to determine the intended context and assessed whether it represented a capacity, capability, or both. Of the 24 definitions of organizational agility, 10 were categorized as capacity; 10 as a future capability; and four provided a mix of capacity and capability. A breakdown of the important components was achieved by analyzing the specific words and meaning within these definitions. As shown in Figure 1, the most repeated components of the definition are "rapid response" and "stimuli is external environment." These are followed closely by "customer driven output," "environment of uncertainty," and "opportunistic outcome." It is important to note that this method of finding common themes amongst definitions suffers from interpretation errors. Interpretation errors are reduced by evaluating each definition element in the context that it was originally provided and making logical contextual adjustments, when necessary, to apply it to the new context. Omissions by the author are also an important source of interpretation error; each omission may be due to purposeful deletion of that element due to its lack of importance in that context. For instance, if an author describes an item as being externally stimulated and others describe it as internally stimulated, further contextual analysis is required for any version that omits internal/external completely. It may be found that an author purposely omitted the element to mean that it is both internally and externally stimulated or that their contextual application does not require further delineation, thus meaning one, the other, or neither. Despite these inherent errors, the cumulative effect of these two error sources is considered insignificant after making the contextual adjustments [2].
The definition provided by Teece et al. [23] includes each of the key components described in Figure 1. Therefore, the following definition is suggested and supported for this field of study: Organizational Agility: "Capacity of an organization to efficiently and effectively redeploy/redirect its resources to value creating and value protecting (and capturing) higher-yield activities as internal and external circumstances warrant" [23]. This definition contains a few "loaded" terms, and thus, it is prudent to provide additional meaning and explanation for key elements of this definition [26].
Efficiently: in a manner that produces desired results with little or no waste. Effectively: producing a decided, decisive, or desired effect. Value Creating: increase in the worth of goods or services. Value Protecting: maintaining the same worth of goods or services. Higher Yield: increase in production from an investment. Warrant: to serve as or give adequate ground or reason for something. It is important to note that this method of finding common themes amongst definitions suffers from interpretation errors. Interpretation errors are reduced by evaluating each definition element in the context that it was originally provided and making logical contextual adjustments, when necessary, to apply it to the new context. Omissions by the author are also an important source of interpretation error; each omission may be due to purposeful deletion of that element due to its lack of importance in that context. For instance, if an author describes an item as being externally stimulated and others describe it as internally stimulated, further contextual analysis is required for any version that omits internal/external completely. It may be found that an author purposely omitted the element to mean that it is both internally and externally stimulated or that their contextual application does not require further delineation, thus meaning one, the other, or neither. Despite these inherent errors, the cumulative effect of these two error sources is considered insignificant after making the contextual adjustments [2].

Related Terms
The definition provided by Teece et al. [23] includes each of the key components described in Figure 1. Therefore, the following definition is suggested and supported for this field of study: Organizational Agility: "Capacity of an organization to efficiently and effectively redeploy/redirect its resources to value creating and value protecting (and capturing) higher-yield activities as internal and external circumstances warrant" [23]. This definition contains a few "loaded" terms, and thus, it is prudent to provide additional meaning and explanation for key elements of this definition [26].
Efficiently: in a manner that produces desired results with little or no waste.
Effectively: producing a decided, decisive, or desired effect.
Value Creating: increase in the worth of goods or services.
Value Protecting: maintaining the same worth of goods or services.
Higher Yield: increase in production from an investment.
Warrant: to serve as or give adequate ground or reason for something.

Related Terms
When examining organizational agility, several related terms consistently appear. It is important to determine the degree of commonality, overlap, and uniqueness of these terms. Organizational resiliency is related to organizational agility, and the two terms are often used interchangeably for one another. There are a significant number of publications that address personal resiliency, however only eight were found that specifically addressed organizational resiliency. Table 2 provides a snapshot of the leading definitions that have been cited in the literature. Table 2. Summary of organizational resiliency definitions [27][28][29][30][31][32][33][34].

Year
Author(s) Definition Recover Advance

1988
Wildavsky (Wildavsky, 1988) The capacity to cope with unanticipated dangers after they have become manifest. X Ability to effectively absorb, develop situation-specific responses to, and ultimately engage in transformative activities to capitalize on disruptive surprises that potentially threaten organization survival.
X X Using the same method as previously described, the key components of organizational resiliency were "response to disruption" (vice opportunity), "recovery outcome" (vice advance), and "reactive" (versus proactive). The definition provided by Lengnick-Hall et al. [34] is the only definition that includes each of these key components. Therefore, the following definition will be applied and supported: Organizational Resiliency: "Ability to effectively absorb, develop situation-specific responses to, and ultimately engage in transformative activities to capitalize on disruptive surprises that potentially threaten organization survival" [34].
Organizational flexibility, robustness, versatility, and adaptability are constructs that also highly relate to organizational agility [2]. Although their work specifically focused on system flexibility rather than organizational flexibility, the research is in the same domain (defense) and is still applicable to this discussion. In their work, the authors reviewed over 200 papers and found 21 relevant definitions for flexibility. Their efforts culminated in an accepted definition through the breakdown of key elements and application of a similar democratic method. Their resultant definition will be applied to this research with a single change. The term system used in their definition was expanded to include the organizations that design, develop, manufacture, and operate the specific hardware solution, and then replaced with the word "organization" to make it applicable to organizations [2].
Organizational Flexibility: "the measure of how easily [an organization's] capabilities can be modified in response to external change." Organizational Robustness: "the measure of how effectively [an organization] can maintain a given set of capabilities in response to external changes after it has been fielded." Organizational Versatility: "the measure of how broadly [an organization's] capability extend in terms of foreseeable and unforeseeable sources of change." Organizational Adaptability: "the measure of how effectively [an organization] can modify its own capabilities in response to change after it has been fielded."

Relationships
The formal definitions put forth in the previous sections leave out their relationship to one another. Agility and resiliency are both organizational characteristics; each describing an organizational response to different stimuli. Agility and resiliency share many of the same key components of their definition. They both require responses to stimuli that may be internal or externally produced and result in an increase (or restoration) in output capability. In manufacturing for instance, that may be the number of units produced, the number of different types of units, the individual unit performance, or even an increase in company profit. In the defense sector, this may manifest itself as speed of production, number of missions supported, decrease in mission time, increase in trained soldiers, etc. Where the definitions of agility and resiliency differ is the type of stimuli. Resiliency is associated with the occurrence of a disruption or issue, which could also be described as a disruption or issue to the status quo, and implies that if the organization does not respond, the output capability will be reduced. Agility is associated with opportunities, where the organization has the opportunity to respond to an event, but failure to do so does not jeopardize the status quo output capability. An organization can possess one, both, or neither of these attributes.
Evaluation of flexibility also shows significant definition overlap with agility and resiliency. Flexibility encompasses the nature of a system (organization) to adapt to change, which is also found in both agility and resiliency. Flexibility differs in that it is determined by the response without a time element. This means that only a single dimension (capability, time, or cost) is required to understand flexibility, while agility and resiliency both require two dimensions (capability and time) to be measured. Any time an organization displays agility or resiliency, it also displays flexibility.

Organizational Agility Framework
Now that we have a working top-level definition of organizational agility, further analysis and breakdown can be accomplished. According to Teece et al. [23] in their 2016 paper, the framework to organizational agility is through a three-step process consisting of sensing, seizing, and transforming, as shown in Figure 2.
Organizational Flexibility: "the measure of how easily [an organization's] capabilities can be modified in response to external change." Organizational Robustness: "the measure of how effectively [an organization] can maintain a given set of capabilities in response to external changes after it has been fielded." Organizational Versatility: "the measure of how broadly [an organization's] capability extend in terms of foreseeable and unforeseeable sources of change." Organizational Adaptability: "the measure of how effectively [an organization] can modify its own capabilities in response to change after it has been fielded."

Relationships
The formal definitions put forth in the previous sections leave out their relationship to one another. Agility and resiliency are both organizational characteristics; each describing an organizational response to different stimuli. Agility and resiliency share many of the same key components of their definition. They both require responses to stimuli that may be internal or externally produced and result in an increase (or restoration) in output capability. In manufacturing for instance, that may be the number of units produced, the number of different types of units, the individual unit performance, or even an increase in company profit. In the defense sector, this may manifest itself as speed of production, number of missions supported, decrease in mission time, increase in trained soldiers, etc. Where the definitions of agility and resiliency differ is the type of stimuli. Resiliency is associated with the occurrence of a disruption or issue, which could also be described as a disruption or issue to the status quo, and implies that if the organization does not respond, the output capability will be reduced. Agility is associated with opportunities, where the organization has the opportunity to respond to an event, but failure to do so does not jeopardize the status quo output capability. An organization can possess one, both, or neither of these attributes.
Evaluation of flexibility also shows significant definition overlap with agility and resiliency. Flexibility encompasses the nature of a system (organization) to adapt to change, which is also found in both agility and resiliency. Flexibility differs in that it is determined by the response without a time element. This means that only a single dimension (capability, time, or cost) is required to understand flexibility, while agility and resiliency both require two dimensions (capability and time) to be measured. Any time an organization displays agility or resiliency, it also displays flexibility.

Organizational Agility Framework
Now that we have a working top-level definition of organizational agility, further analysis and breakdown can be accomplished. According to Teece et al. [23] in their 2016 paper, the framework to organizational agility is through a three-step process consisting of sensing, seizing, and transforming, as shown in Figure 2. Sensing is the identification of technological opportunities and is critical if an organization is to ever attempt to capitalize on them. "Generative-sensing capabilities involve undertaking actions to proactively create hypotheses about the future implications of observed events and trends, and testing these hypotheses to grease the pathways for new products, services, and business models" [3].
Scenario planning and what-if analysis (i.e., development planning within the DoD) are typical sensing techniques. Sensing is more than predicting future customer desires; it also includes the synthesis of different ideas, processes, and technologies to form new products that provide value to the consumer. Existing organizations tend to focus on existing ideas and processes, whereas new entrants are often more poised to develop new combinations and technological innovation [35,36]. Within each of those organizations, middle-level management is the most acute at splicing together different ideas and technologies, and executive level management is better poised to understand the changing customer desires [37].
Seizing is the implementation of new systems, processes, or services. It is the first step that requires the sizable expenditure of resources, as investments in development are often required [23]. The total amount of uncertainty has been reduced, with a portion being converted into quantifiable risk. An organization must be poised to seize opportunity, as "addressing opportunities involves maintaining and improving technological competences and complementary assets and then, when the opportunity is ripe, investing heavily in the particular technologies and designs most likely to . . . acceptance" [23]. In the business world, this often involves having a stockpile of cash reserves, equipment and/or expertise, while this manifests in the DoD as trained personnel, stockpile of equipment, allies, the budgeting processes, and a decision process that evaluates and welcomes opportunities.
Transforming is the restructuring of an organization to capitalize on a new technology. The newest methodology to do this is through a practice known as "build-measure-learn" where a minimum viable product is produced, allowing the company to release it, learn from their successes and mistakes, and quickly improve the product [23]. Similarly, the DoD has recently created an acquisitions model with similar characteristics known as rapid prototyping. This, when paired with creating small "startup" units within the organization to manage the new technology, allows an organization a reduction in risk when developing a new technology, while remaining poised to capitalize on those that succeed. Each transformation has a cost that must be overcome each time an organization attempts to take advantage of an opportunity. This transformation cost represents the non-value-added effort required for the organization to transition from one to state to another. For organizations with a high transformation cost, this can be seen as an agility inhibitor.

Existing Agility Measurement Methods
Despite the desire for organizations to become agile, the ability to measure agility has remained elusive. The difficulty arises when trying to create a measure that is both general enough to apply to multiple industries, yet specific enough to capture the important essence of each particular industry [16]. To address this, most measures of agility to date are domain specific. Further, agility joins other important metrics such as morale, happiness, satisfaction, justice, and quality, in that it is not directly measurable. A latent construct, which is where a variable is inferred through a model from other variables that are more readily observed, is required [38]. To date, there have been several attempts at measuring agility. A summary of these methods follows.

1.
The two-dimensional dichotomy is the most common method used to measure organizational agility. It frequently manifests itself in the form of magnitude of variety/change and the response time/rate [39]. These variables exist with a degree of dichotomy; the actions required by an organization to increase the magnitude of variety of services or products is often contradictory to a firm's ability to increase efficiency and reduce their response time [40]. The magnitude of variety/change attempts to capture an organizations current capability of interest, and to quantify their change in that domain. For instance, for a smart phone manufacturer, it may be increased production, greater features on a device, a greater variety of devices produced, or a new method to reduce the cost to produce each item [40]. The response time/rate variable is meant to capture the temporality of the change in a suitable unit of time, such as days, months, per year, or per cycle [40]. Both dimensions are applicable across multiple industries, however they must be calibrated for their respective industry.

2.
First-order models that calculate agility by relying on the magnitude of variety and response rate have been developed by multiple authors [41,42]. These first order models often lack support and applicability across different industries (domains). More specifically, no models have been developed to apply to the defense sector.

3.
Agility curves were developed and presented by [39]. The agility curves have significant meaning: two points on the graph can result in the same agility rating, and there is an inherent tradeoff between the magnitude of variety change and rate of variety change. Both of these notions are aligned with the argument of dichotomy between the dimensions. This model is supported within the academic community; however, it lacks a simple, repeatable method to measure the magnitude of variety and response rate, and the scale can be difficult to determine and is thus limited in its actual implementation.

4.
Comprehensive agility measurement tool (CAMT), developed at Old Dominion University [16], has proven industry agnostic. The tool relies on ten "agility enablers" to measure agility on a scale of 1 to 5 and an analytical hierarchy process (AHP) to ensure that it can be effectively applied to a multitude of industries. Starting from the set of 41 agility enablers found by [43], the survey administrator selects the ten most relevant factors for the given domain and assesses them utilizing a 5-point Likert scale. After applying a weighted average to each of the ten areas, a weighted agility measure is calculated. Although CAMT uses a mathematical model, it is highly subjective due to the administrator's selection of the ten relevant factors, and the weights applied to each agility enabler. The subjectivity required within CAMT has inhibited its overall support and application.

5.
Key Agility Index (KAI) is a method developed by Lomas et al. to measure design process agility by assessing the product development process and making the case that each product process provided a narrow glimpse of the overall organization's agility. They developed the Key Agility Index, which is the ratio of "time taken to complete change related tasks and time taken to complete the whole project" [44]. This method has high internal validity within a domain, but the authors warn against comparison between different market sectors. Further, this model fails to take into account other factors, such as an effective systems engineering plan. For instance, a product with a poor quality systems engineering plan will likely require a greater number of changes and greater overall variability in the time required to complete change related tasks [44].
Each of these methods provides a different approach to measure organizational agility but currently lacks application within the defense sector. Further, there are no measurement methods that tie directly to the definition of OA that we have adopted.

Research Objectives
The objective of this manuscript is to isolate the variables needed to measure organizational agility (OA) in military organizations, allowing for the future development of a suitable method to measure OA without the need to interact with outside organizations. Once captured, these variables will form the necessary and common foundation needed by researchers to develop a method to measure OA. The OA measure, in-turn, will allow organizations the metrics needed to evaluate their internal agility and provide organizations the tools necessary to efficiently and effectively re-allocate resources in their ongoing quest to increase their OA.

Developing a New Organizational Agility Measure
Measures of performance are present in nearly all aspects of life. Their contribution to individual and organization performance is undeniable, and their mere existence often causes changes in behavior. More specifically, measures of performance provide a means to quantify success and in turn contribute to the development of effective incentive structures. When accurately and effectively measured, they can be used to steer performance to achieve higher level objectives, ultimately changing behaviors. Unfortunately, most fields outside of the financial sector struggle to obtain suitable measures that are valid and reliable [45]. Latent constructs are developed when a variable of interest cannot be observed or measured directly, and thus, measurement is achieved via a theoretical relationship between that variable of interest and other, more directly, measurable indicators, known as factors.
The work completed by Colquitt [46] in summarizing a method to utilize survey research to create a latent construct, and his subsequent application to develop an organizational justice measure, can be similarly applied to develop a measure for organizational agility. Utilizing the assumption that there is a set of factors that can be used to measure organizational agility, the next step is to identify any relevant factors.

Factors Related to Organizational Agility
Many researchers have attempted to capture the important characteristics of organizational agility. By collecting the sets of characteristics developed by other researchers, a more complete single set of characteristics was created. The process was to collect all prospective characteristics that could be used to measure agility, and then to systematically remove duplicates and non-relevant items. Since agility is highly related to constructs such as flexibility, rapidness, resiliency, and robustness, any characteristic used in their descriptions were also collected.
Utilizing a 3-round Delphi study designed to develop the framework for a survey questionnaire on leanness and agility, Kuruppalil [43] identified the top 45 agility indicators for job shops from 14 different domains. Yusuf et al. [11] studied manufacturing agility and found 32 key attributes comprised within 10 different domains, which was later reduced to seven a few years later [11,47]. Research conducted by Lepore et al. [48] that focused on military rapid development projects found 11 unique attributes by utilizing in-person interviews. Table 3 provides a summary of agility characteristics offered by these publications.  Each of these characteristic sets were created to fully encompass organizational agility, meaning that each of these sets are believed to be comprehensive and complete, albeit in their respective domains. With the sets provided by Kuruppalil [43] and Yusuf et al. [11], both originating in the manufacturing domain, one would expect there to be significant overlap in sets. Further, the characteristic set provided by Lepore et al. [48] provides a well-needed bridge into the military domain. By combining the three sets into a single set, it is reasonable to believe that (1) the new set will be larger than each of the individual sets, (2) the new set will have a greater chance of encompassing the factors necessary to develop a latent construct, and (3) there will be redundancies within the new set. In most cases when combining datasets, redundancy is relatively easy to identify and eliminate. In this case, however, redundancies are difficult to recognize due to the varied wording used to describe each characteristic. The Q-sort method was used to compare, combine, and reduce redundancies in these sets.

The Q-Sort Method
Q-sort is "a method of assessing reliability and construct validity of questionnaire items being prepared for survey research" [49]. First developed and published by Catell [50], the Q-sort method was one of the six correlation methods (O, P, Q, R, S, and T). The Q-sort method was further refined by Stephenson [51] and Block, J. [52] into the incarnation that is used today. It is an iterative process where the level of agreement between judges is measured and used to determine overall construct validity [49][50][51][52][53].
The procedure to conduct a Q-sort is as follows: 1. Collect items to be sorted. These items are expected to be a sample from the entire population of items that could be used.

2.
Select number and capacity of judges. One of the most useful features of the Q-sort method is the limited experience and training that is required of the judges to conduct the sorting. Judges should be knowledgeable in the domain specific to the items, but do not need any formal experience in the Q-sort method itself. The minimum number of judges is two, however the benefit of having additional judges beyond two is often quickly outweighed by the level of disruption it causes when calculating Cohen's Kappa and the level of agreement. For these reasons, two judges are often preferred.

3.
Apply a suitable construct in which the judges can sort the items. This construct may be developed in advance or by the judges themselves. It is recommended that the construct include an "other" category for items that are difficult to fit into a single category.

4.
Judges sort the items independently. Methods to ensure independence include keeping each judge out of view of the other, sort via a computer database, or having the items to be sorted in a different, random order for each judge.

5.
Calculate Cohen's Kappa and the agreement ratio. To calculate the agreement ratio, a table that utilizes the number of items for each category is constructed. Figure 3 provides a generic setup for judges (most common); a similar three-dimensional model can be created if three judges were used.
judge out of view of the other, sort via a computer database, or having the items to be sorted in a different, random order for each judge. 5. Calculate Cohen's Kappa and the agreement ratio. To calculate the agreement ratio, a table that utilizes the number of items for each category is constructed. Figure 3 provides a generic setup for judges (most common); a similar three-dimensional model can be created if three judges were used. Converting Figure 3 into percentages can be done by dividing each table cell by N, resulting in Figure 4.  The agreement ratio is then calculated as: Cohen's Kappa is calculated by first determining the chance of agreement and then removing it from the total number of actual agreements. Chance agreements are calculated by multiplying the cross row-column totals, as: From there, the total number of actual agreements is also calculated, as: The difference between the Number of Actual Agreements and Total Chance of Agreement, standardized for the maximum possible value, which is known as Cohen's Kappa, can then be calculated as: There is no agreement on a minimum acceptable Cohen Kappa. Landis and Koch published a detailed guideline in their 1977 work, where they provided the following recommendation [54]: Cohen's Kappa is calculated by first determining the chance of agreement and then removing it from the total number of actual agreements. Chance agreements are calculated by multiplying the cross row-column totals, as: From there, the total number of actual agreements is also calculated, as: The difference between the Number of Actual Agreements and Total Chance of Agreement, standardized for the maximum possible value, which is known as Cohen's Kappa, can then be calculated as: There is no agreement on a minimum acceptable Cohen Kappa. Landis and Koch published a detailed guideline in their 1977 work, where they provided the following recommendation [54]: Using the guidelines from Landis and Koch, a minimum Kappa of 0.61, representing "substantial agreement," was used.

Applying the Q-Sort Method to Organizational Agility
The Q-sort method was applied to the agility characteristics already described. The ultimate goal was to determine which, if any, characteristics were redundant in the set. In accordance with the recommendations by Ozer [53], two judges were used. Both judges had backgrounds representative of the expected survey respondents' that would be used later in this research but possessed minimal knowledge on the Q-sort method. The judges were given 1 h of training and expectations briefing. The two-day evaluation was performed in 2019. The procedure required a two-round Q-sort method, each round further delineating and categorizing each characteristic [53]. Round 1. Both judges were given the complete set of items from Table 3 (N = 88) and were asked to categorize each item. Previous research on the OA framework by Teece et al. [23] resulted in three categories for OA characteristics, including sensing, seizing, and transforming. These three categories, along with their descriptions provided by Teece et al. [23], were used to form the bins for the first round. A brief description of these categories was given to each judge to better align their meaning against that of the original authors, and to reduce any pre-conceived notions. Each of the items were written on a 3 × 5 index card, and subsequently shuffled (randomized) for each judge to ensure independence. Once the judges were both complete, the cards were sorted and the agreement ratio calculated, as shown in Figure 5. The dataset was then normalized (divide by N) and Cohen's Kappa was calculated to be 0.74, as shown in Figure 6. This met the criteria of 0.61, ("substantial agreement"), and the process was advanced to round 2. The two-day evaluation was performed in 2019. The procedure required a two-round Q-sort method, each round further delineating and categorizing each characteristic [53]. Round 1. Both judges were given the complete set of items from Table 3 (N = 88) and were asked to categorize each item. Previous research on the OA framework by Teece et al. [23] resulted in three categories for OA characteristics, including sensing, seizing, and transforming. These three categories, along with their descriptions provided by Teece et al. [23], were used to form the bins for the first round. A brief description of these categories was given to each judge to better align their meaning against that of the original authors, and to reduce any pre-conceived notions. Each of the items were written on a 3 × 5 index card, and subsequently shuffled (randomized) for each judge to ensure independence. Once the judges were both complete, the cards were sorted and the agreement ratio calculated, as shown in Figure 5. The dataset was then normalized (divide by N) and Cohen's Kappa was calculated to be 0.74, as shown in Figure 6. This met the criteria of 0.61, ("substantial agreement"), and the process was advanced to round 2.    Round 2. The categories used in round 1 (seizing, sensing, transforming) were each broken down into subcategories. The judges were allowed to select the subcategories via a discussion and consensus process amongst themselves. Although the judges were allowed to select from 2-5 subcategories, each of the subcategory selections also resulted in three subcategories. From there, the Round 2. The categories used in round 1 (seizing, sensing, transforming) were each broken down into subcategories. The judges were allowed to select the subcategories via a discussion and consensus process amongst themselves. Although the judges were allowed to select from 2-5 subcategories, each of the subcategory selections also resulted in three subcategories. From there, the same process as described in the previous round was repeated. The hierarchical structure and results of round 2 are shown in Figure 7. It is important to note that the first time through in the category of transforming, the judges resulted in a Cohen's Kappa of 0.498. This was significantly lower than the goal of 0.61, so a mediation round occurred. During this mediation round, each judge was given 60 s to discuss the disparate items. Following the time limited discussion, each judge then re-scored the item in secret. After the second attempt within the "transforming" category, the Cohen's Kappa was increased to 0.914. The mediation process had been pre-determined and agreed upon by the judges before the start of the sorting, however extreme caution should be taken when employing such a technique, as it may invalidate the assumption of independence. In this case, it was determined the breach of independence was preferred over proceeding with a Cohen's Kappa of 0.498.
At this point, the Q-sort method was complete in its entirety. A final round of discussions was completed to determine which, if any, items were redundant in nature. The judges were given the items, one subcategory at a time (of the nine total subcategories), and they searched for redundancies. Open discussion and deliberation was encouraged, and it took both judges to agree before a redundancy was declared. In most cases, redundancy were between two items, however a few occurrences of three-item redundancy did occur. In total, 24 redundant items were removed from the list.
to discuss the disparate items. Following the time limited discussion, each judge then re-scored the item in secret. After the second attempt within the "transforming" category, the Cohen's Kappa was increased to 0.914. The mediation process had been pre-determined and agreed upon by the judges before the start of the sorting, however extreme caution should be taken when employing such a technique, as it may invalidate the assumption of independence. In this case, it was determined the breach of independence was preferred over proceeding with a Cohen's Kappa of 0.498. At this point, the Q-sort method was complete in its entirety. A final round of discussions was completed to determine which, if any, items were redundant in nature. The judges were given the items, one subcategory at a time (of the nine total subcategories), and they searched for redundancies. Open discussion and deliberation was encouraged, and it took both judges to agree before a redundancy was declared. In most cases, redundancy were between two items, however a few occurrences of three-item redundancy did occur. In total, 24 redundant items were removed from the list.

Results and Discussion
In the ongoing effort to identify the characteristics of an agile organization, this research accomplished three important objectives.
First, through the analysis of the available OA definitions, an acceptable, commonly applicable definition was found that can be utilized in the defense sector and for future OA measure development. This definition was found by evaluating 24 different definitions that have been previously offered and adjusting them for the relevant context. By disassembling them each of them into their basic components, analysis was completed to determine the relevance of each piece in the greater context of the OA. A definition that contained the most important components, while purposely omitting contentious items, was selected and supported. This effort culminates in a single, commonly accepted definition that can be used by organizational behavioralists, researchers, and practitioners from here forward.

Results and Discussion
In the ongoing effort to identify the characteristics of an agile organization, this research accomplished three important objectives.
First, through the analysis of the available OA definitions, an acceptable, commonly applicable definition was found that can be utilized in the defense sector and for future OA measure development. This definition was found by evaluating 24 different definitions that have been previously offered and adjusting them for the relevant context. By disassembling them each of them into their basic components, analysis was completed to determine the relevance of each piece in the greater context of the OA. A definition that contained the most important components, while purposely omitting contentious items, was selected and supported. This effort culminates in a single, commonly accepted definition that can be used by organizational behavioralists, researchers, and practitioners from here forward.
Second, utilizing three highly researched and distinct sets of OA characteristics, each representing a different domain or industry focus, a larger, more encompassing set was created. Since each of the original characteristic sets found in the literature were the result of extensive studies, each were expected to contain all of the needed characteristics to construct a measure for OA. Further, each of these sets were the result of a different research method and/or domain, thus resulting in different, albeit similar, outcome sets. The aggregation of characteristic sets, by its very nature, greatly decreased the likelihood that a particular important characteristic was missing, as it would have to have been missing in all three of the original researcher's lists. Thus, a more complete characteristic set was created.
Third, characteristics from the aggregated set contained some redundant and overlapping terms. By using the Q-sort method, each characteristic was systematically analyzed against all other items in the set. Twenty-four characteristics were selected for removal from the set, reducing the set by 27%. The reduced set offers significant advantages over the full, aggregated set. During future efforts in this area, researchers can more efficiently focus their attention, and if a survey is used, respondents will be better suited to answer the questions and have less errors due to concept overlap. Table 4 contains the final, reduced set of OA characteristics. Together, these three objectives help in establishing a common understanding of OA. Further, they form the necessary foundation to establish a method to measure, and ultimately improve, OA.

Significance
The result is a suitable definition for organization agility applicable to military organizations and a list of potential associated characteristics that summarize related research to date. This groundwork establishes the foundation to conduct a multi-organization study to further refine the characteristic list and ultimately develop a method to measure organizational agility. With these results, practitioners can identify the important characteristics related to OA and can refocus internal training and resources to improve their organization in terms of OA. The foundations of OA developed here are the important bridge and re-invigoration needed in the ongoing study of OA and the ultimate goal of fully measuring it.

Limitations and Future Work
This research encountered several limitations. First, although the literature review found a significant number of related publications, recency was an ongoing issue. The research was limited to publications in English, and thus, the probability that numerous non-English publications were omitted is quite high. Second, the lists of characteristics used were from the manufacturing domain and a study of the rapidness of defense acquisitions. No direct study of OA-related characteristics, specifically relating to OA in the military sector, was found. Thirdly, only two judges with relevant experience were available for the two-day Q-sort. Further, adding a third or fourth judge would have likely extended the process, which would have caused additional availability issues with the existing judges.
Future work in this arena is envisioned to include (1) further literature search in using additional search techniques, databases, and languages; (2) re-accomplishing the Q-sort with new and possibly more judges to compare against the existing results; (3) research to solicit additional OA related characteristics unique to military organizations, and (4) development of a survey to collect data on OA using the reduced set of characteristics.

Conclusions
There is a continuous need for organizations to become agile in order to survive and succeed amongst their peers. A method to accurately measure organizational agility within the DoD has yet to be fully developed. Through a literature review, a suitable and formal definition for organizational agility was found and support confirmed. An initial set of related characteristics, which can be used to develop a latent construct, was discovered and analyzed. Utilizing the Q-sort method, redundant characteristics were eliminated resulting in 64 remaining characteristics that will be used to develop the necessary survey questions to continue this research.