Measuring Knowledge Management Performance in Organizations: An Integrative Framework of Balanced Scorecard and Fuzzy Evaluation

Abstract

As an important part of knowledge management (KM), the KM performance evaluation tries to find out the key factors restraining the enhancement of the enterprises' performance. This paper investigates the feasibility of the balanced scorecard (BSC) method in enterprise knowledge management and then proposes a simplified and applicable performance evaluation model based on the BSC approach. Finally, fuzzy comprehensive evaluation (FCE) is used to evaluate the effectiveness and applicability of the proposed model. The result shows that the model is useful for evaluating the performance of KM in enterprises.

1. Introduction

Organizations have undergone the transformation of managing tangible goods to knowledge assets [1]. How to effectively and efficiently engage in knowledge management (KM) has become an essential problem for organizations to solve in modern enterprise management and innovation [2,3,4].

KM strategies adopted and implemented by an organization significantly influence its KM initiatives [5], which need to be evaluated regarding their performances so that they can align with the organization’s strategic goals [6].

KM performance evaluation, as an indispensible component of KM, plays a pivotal role in the entire process of KM [7]. An effective evaluation of KM performance will allow managers to understand the deficiencies in their management practices, identify the key elements impacting organizational development, and provide theoretical evidences for continuous improvement.

Although its importance is unanimously accepted by business practitioners and academic researchers, there is a lack of a consensus on the standardized method to measure the performance of KM in organizations. Various models (e.g., [8,9,10]) have been proposed to help organizations design feasible solutions to evaluate the effectiveness of their KM strategies. However, prior models are either too theoretically oriented or difficult to implement. Our research addresses this gap by constructing a KM performance evaluation framework based on balanced scorecard (BSC) and fuzzy comprehension methods, which simplifies the assessment and implementation process.

The rest of the paper proceeds as follows. The next section reviews prior literature with an emphasis on the available models for evaluating KM performance in organizations. Section 3 presents our KM performance evaluation framework based on BSC. Section 4 details the construction of the evaluation model by using the fuzzy comprehensive evaluation method. Section 5 demonstrates the applicability of our evaluation model with a numerical example. The last section concludes the entire paper.

2. Literature

This section reviews prior literature. We first summarize related research based on the following two research streams: (1) varieties of KM performance evaluation methods and (2) BSC as an approach for evaluating KM performance. Then we highlight the focus and contributions of our study.

Many unique perspectives about KM performance evaluation have been raised by different organizations. However, due to the complexity of KM, there is still a lack of a consensus on the KM performance evaluation system. In 1996, the Organization for Economic Co-operation and Development (OECD) proposed a framework containing major indicators for the knowledge-based economy to measure knowledge inputs, stocks and flows, outputs, networks, and learning from a micro perspective (OECD, [11]). Based on its KM maturity model, the American Productivity and Quality Center (APQC) established a KM capability assessment tool to assess an organization’s KM competencies including four categories: strategy, people, process, and contents and information technology (APQC, [12]).

A variety of methodologies have been applied in KM performance evaluation by scholars (see

Table 1. A summary of the methodologies for KM performance evaluation in recent literature.

Paper	Representative Methodologies of KM Performance Evaluation
Lee et al. ([13], 2005)	Associating knowledge management performance index (KMPI) with three financial indicators (stock price, price earnings ratio, and research and development expenditure)
Kuah et al. ([14], 2012)	Integration of data envelopment analysis (DEA), Monte Carlo simulation, and genetic algorithm (GA)
Yin and Fai ([8], 2014)	Integrated knowledge management (IKM) model
Chen and Fong ([15], 2015)	Structural equation modeling (SEM) and system dynamic (SD) simulation
Lee and Wong ([16], 2015)	Survey instrument in small and medium companies
Wang et al. ([17], 2016)	Synthetic evaluation method by using triangular fuzzy numbers

for a brief summary). For instance, Lee et al. [13] developed a framework of the knowledge management performance index (KMPI) including five components (knowledge creation, accumulation, sharing, utilization, and internalization) to assess the efficiency of the knowledge circulation process (KCP) within organizations by associating KMPI with three financial indicators (stock price, price earnings ratio, and research and development expenditure). Kuah et al. [14] developed a KM performance evaluation model by integrating several quantitative methodologies including data envelopment analysis (DEA), Monte Carlo simulation, and genetic algorithm (GA). Incorporating the total quality management and business excellence models, Yin and Fai [8] derived an integrated knowledge management (IKM) model to evaluate how KM performance contributes to organizational goals. Chen and Fong [15] constructed a KM performance index with the structural equation modeling (SEM) and system dynamic (SD) simulation approaches. Lee and Wong [16] designed a survey instrument for KM performance measurement in small and medium companies. Combining subjective and objective measures together, Wang et al. [17] created a synthetic evaluation method by using triangular fuzzy numbers to estimate the indexes they propose and a group support system (GSS) to execute the process of KM performance evaluation. In summary, the KM performance evaluation methods from the research between 1995 and 2004 can fall into one of the following eight categories of analysis: qualitative, quantitative, financial indicator, non-financial indicator, internal performance, external performance, project-orientated, and organization-orientated [18]. Additionally, six new categories (traditional, advanced, deterministic, stochastic, general result–oriented, and specific result–oriented) have been recently proposed for KM performance measurement [7].

Among many approaches for KM performance evaluation, BSC appears to be a promising one as KM naturally fits in the learning and growth category of the BSC framework [19]. Since Kaplan and Norton [20] proposed the concept of BSC as a strategic management system, it has been widely used for studying the performance evaluation of different disciplines, such as customer relationship management (e.g., [21,22]), human resource management (e.g., [23,24]), information and communication technology (e.g., [25,26]), and supply chain management (e.g., [27,28]). In the KM area, many studies have started to demonstrate the applicability of BSC for KM performance evaluation. For instance, De Gooijer [29] defines KM as the actions supporting collaboration and integration and establishes two models for measuring KM performance and behaviors. The proposed KM performance scorecard model links KM objectives with the four major elements of BSC in organizations proposed by Kaplan and Norton [20]. Kao et al. [9] utilize the BSC approach to measure the performance of an online knowledge community through four aspects: customer, internal process, learning and growth, and performance. Lin [10] develops a model to investigate the impact of different dimensions of KM (e.g., organizational memory, knowledge sharing, and knowledge absorption, etc.) on the four outcomes of BSC and finds that knowledge sharing can be used to predict the performance of internal processes and knowledge absorption can indicate and improve customer satisfaction levels. Valmohammadi and Ahmadi [30] study the impact of KM practices on organizational performances through a BSC approach and demonstrate that the impact is significant only on the dimension of growth and learning and insignificant on the other three dimensions of BSC outcomes. Combining the analytic network process (ANP) with a BSC, Hu et al. [31] propose a framework to evaluate the performance of knowledge assets via the value of three indicators: labor, technology, and utilization.

The research findings from prior literature demonstrate the inconsistencies among the frameworks of KM performance evaluation. Some of the evaluation frameworks are too abstract, have no systematic structure, and contain overlapping appraisal items, whereas others are not fully operational, lack objective evaluation results, and are not comprehensive. Our research proposes a KM performance evaluation system by integrating a fuzzy evaluation method. In particular, we incorporate the concept of fuzzy mathematics proposed by Zadeh [32] and Goguen [33] to simplify the assessment of relevant indicators of BSC and generate intuitive evaluation results.

3. A Framework of KM Performance Evaluation Indicators Based on BSC

This section proposes a framework of evaluating KM performance in organizations based on a BSC approach. We first justify the applicability of BSC theory in evaluating KM performance, and then map specific KM processes with BSC outcomes. In addition, we demonstrate the causal relationship between KM and BSC, which is used to establish a formal framework consisting of the indicators to measure KM performance.

3.1. BSC for KM Performance Evaluation

BSC is a multi-purpose performance evaluation system that can transform the strategy and goals of an organization and its units into a set of diversified and inter-related objectives, and can then decompose the objectives into several indicators [20]. This method overcomes the drawbacks of using the single financial measures for performance management by integrating the four perspectives (finance, clients, internal business processes, and learning and growth) to inspect the value creation. BSC focuses on not only the current financial status of an organization, but also the motives of its long-term financial performance, so that the organization can identify the factors to improve its future performance and methods to create new values through investing in clients, suppliers, employees, and technological innovation.

KM emphasizes ways to improve organizational learning, knowledge sharing, and innovative capabilities; re-organize internal business processes; improve organizational structures so as to streamline knowledge processes; become more responsive to clients’ needs; and enhance core competitive advantages, which can all be reflected through the maximal benefits that an organization can obtain. The primary evaluation objects of KM are knowledge and the knowledge-based intangible assets, so its evaluation method should be balanced between objective and subjective, qualitative and quantitative ones. Therefore, the evaluation of KM performance needs a tool that can assess both tangible and intangible benefits, and overcome the problems of time lag, partiality, and poor indicators, which can all be attained by the four indicators of BSC. In summary, BSC theory serves as an appropriate instrument for evaluating enterprise KM performance as it provides a brand new and comprehensive perspective by setting up concrete indicators from finance, clients, internal business processes, and learning and growth aspects.

3.2. The Mapping between KM and BSC

KM can be considered as a collection of processes including knowledge acquisition, sharing and transfer, application, and creation [34]. The purpose of KM is to maximize the value creation for an organization through consolidating its client and human capitals. Therefore, the KM performance evaluation framework should be aligned to the objectives of enterprise KM to help organizations maximize the benefits of their KM initiatives. Since BSC incorporates finance, clients, internal business processes, and learning and growth and an enterprise’s value creation can be reflected through these four aspects, we believe that the mapping between KM and BSC can be established as shown in Figure 1.

Representing the interests of the stakeholders, the indicators from the finance aspect can be considered as the originator and destination of the other three aspects and thus infer an organization’s short-term and long-term strategies and whether their implementations can contribute to improving the organization’s final operational results. Therefore, although KM performance may vary for different organizational strategies according to the requirements for long-term or short-term benefits, it is essentially aligned with an organization’s finance performance. The purpose of knowledge application is to improve an organization’s management efficiency and knowledge workers’ productivity levels so as to gain competitive advantage, which can be ultimately reflected through the measurable financial indicators. Therefore, knowledge application can be mapped to the finance aspect.

The client aspect indicates how an organization responds to external changes, which can be demonstrated through the quality, performance, and support of the organization’s products and services. Through knowledge application (e.g., implementing a CRM system), organizations can improve the quality, performance, and after-sale support of its products and services. In addition, through knowledge creation, organizations can fulfill their missions of engaging in continuous innovation of their technologies and services so as to attract more customers. Therefore, knowledge application and creation can be mapped with the client aspect.

The aspect of internal business processes primarily reflects the current priority strategy that facilitates various business processes to satisfy the needs of clients and other stakeholders. Internal business processes refer to those activities that allow organizations to create value-added products or services, particularly those operational processes that can improve customer satisfaction levels and have the most impact to the organizations. KM plays a very important role in improving such business processes for organizations. For instance, through knowledge application, managers can better apply knowledge in decision-making so as to push forward the policy implementation more effectively. Knowledge transfer and sharing promotes the sharing and disseminating of the best practices in organizations by creating new channels for employees to obtain and share knowledge. In addition, knowledge acquisition allows organizations to obtain knowledge from external sources to continuously improve and streamline their internal business processes. Therefore, an organization’s capability of knowledge acquisition, transfer, and sharing demonstrates the smoothness of its internal processes, so the internal business process aspect corresponds to knowledge acquisition, storage, transfer, sharing, and application.

The learning and growth aspect mainly deals with the ways for organizations to develop a culture that can sustain its adaption, innovation, and growth. If an organization wants to sustain its development, its employees must continuously learn, grow, and innovate. Knowledge application and creation enhance an organization’s learning and growing ability. In addition, knowledge creation can be regarded as the achievements of employees’ learning and the manifestation of their growth. Therefore, the learning and growth aspect matches knowledge application and creation.

3.3. The Causal Relationship between KM and BSC

One of the key objectives of BSC is to articulate an organization’s developing strategies, which can be achieved through investigating the major factors in the causal relationship. As Figure 2 demonstrates, the four aspects of BSC are not isolated, but inter-connected. Although the four BSC aspects are influential to each other, we focus on the inter-connectivity of these four aspects that can be facilitated through various aspects of KM processes. For instance, the innovative and integrative nature of KM processes contributes to the learning and growth aspect of BSC, resulting in the improved internal business processes along with the efficient operation of KM processes.

3.4. The Framework of Indicators

Based on the corresponding relationship between KM and BSC from the previous section, we synthesize BSC indicators and the enterprise information performance evaluation indicator framework, and then summarize the KM performance evaluation indicator framework (see Figure 3). We next explain the rationale of selecting the indicators in our framework.

As we mentioned earlier, the ultimate strategic goals of KM can all be reflected through its financial benefit. Therefore, we choose the following three major financial indicators to measure this aspect: sale growth rate, net asset rate of return, and total asset rate of return. These three indicators demonstrate an organization’s profiting ability, asset utilization efficiency, and its potential to connect with its clients.

From the client aspect, the successful implementation of an organization’s client management programs as one of its KM initiatives can not only satisfy and retain customers, but also acquire new customers through the word-of-mouth effect so as to expand market shares. Therefore, we use the customer retention rate, the new customer acquisition rate, and the market share as the three indicators to evaluate the client aspect.

The internal business process is normally evaluated through indicators such as data accuracy, process standardization, system integration, and product quality. However, these indicators can all be demonstrated through an organization’s efficiency and efficacy improvement from the perspective of KM performance. Therefore, we simplify them into the two indictors of per capita management cost and per capita profit.

An organization’s innovative capability and its employees’ learning abilities for skill improvement are closely related to its value creation. Employees must continuously learn, grow, and innovate for an organization to sustain its development. Knowledge application and creation reinforce an organization’s learning and growth ability to engage in innovation. The learning and growth of an organization can be exhibited through two dimensions: (1) its employees’ capabilities and (2) organizational innovation. The employee dimension incorporates the opportunity and investment for employees to obtain training, to improve their abilities, and to enhance their satisfaction levels in organizations, which can be measured through the three indicators of per capita training investment, per capita profit generation rate, and employee turnover rate, respectively. The other dimension, organizational innovation, includes many indicators such as perception, organization, market, and technology innovation, which all tie into knowledge application and can be ultimately derived from knowledge creation. However, because these indicators are difficult to quantify and are eventually shown as product innovation, we choose to use the new product selling rate to measure these indicators through a single gauge of product innovation based on the principle of simplicity.

4. Evaluation Model

Following upon our proposed KM evaluation framework, we next illustrate the steps to implement the framework by quantitatively measuring the indicators and generating an overall result with the fuzzy comprehensive evaluation method.

4.1. Evaluation Object Factor Sets

We create two levels of evaluation factors: U = (U1, U2, U3, U4) and U1 = (u11, u12, u13), etc. More levels can be added if necessary. In this framework, Ui (i = 1, 2, 3, 4) and u1j (j = 1, 2, 3) are the actual factors in their individual levels.

4.2. Comment Sets

The comment sets are those conclusive comments from experts on evaluation objects. There cannot be too many levels of the comment sets as it will be difficult to control the standards, nor can the number of levels be too few, as it will be difficult to differentiate the levels. In order to comprehensively and accurately reflect the actual situation, the expert group consists of experts, top managers, and users in each field. After an expert group is created, the comment set can be confirmed, which is denoted as V = {v₁, v₂, v₃, v₄} = {“Excellent”, “Good”, “Normal”, “Poor”}.

4.3. Weighted Distribution of the Indicators

The method of the analytical hierarchical process (AHP) is applied to create the assessment matrix and calculate the distribution of the weights.

First, we construct the assessment matrix and measures. The expert survey evaluation method can be used to obtain the comparison between an indicator in a level and that in the previous level, resulting in the assessment matrix $B={(b_{ij})}_{nn}$ at each level.

Second, we evaluate the consistency of the matrix according to the AHP method. The formula $CR=CI/RI$ is used for evaluation, in which CR is the consistency ratio of the evaluated matrix. CI is the consistency index that can be calculated from $CI=({\lambda }_{\max }-n)/(n-1)$, which measures the degree or deviation of the consistency level, and RI denotes the random index with a list of pre-defined consistency-level values such as 0, 0, 0.58, 0.9, 1.12, 1.24, 1.32, 1.41, 1.45, corresponding to rank n of the evaluation matrix from 1 to 9.

Third, we calculate the weight vector of a single evaluation matrix. There are two simple methods for computing weight vectors: summation and root. We use the summation method based on the improvement of standardizing each column vector of the evaluation matrix before calculating the weight vector.

Fourth, we calculate the overall priority vector through integrating all the evaluation matrices by taking into account the priority vector for each criterion and the priority vector between all criteria.

4.4. Determine the Membership Value of Indicators and Create Fuzzy Relationship Matrix R

We use the concept of membership functions, first proposed by Zadeh [32], to create the fuzzy relationship matrix for each dimension of the BSC outcomes. A membership function generalizes how each point in an input space corresponds to a membership value, which is a concept distinct from probabilities.

4.4.1. Membership Value of Positive Indicators

The positive indicators are those that will contribute positively to the overall goal when their values increase. Their models are:

$$R_{ij1}=\{\begin{array}{ll}\begin{array}{l}1\\ (u-x_2)/d\end{array}\hfill & \begin{array}{l}u\ge b\\ x_2\le u\le b\end{array}\hfill \\ 0\hfill & u<x_2\hfill \end{array}, R_{ij2}=\{\begin{array}{ll}\begin{array}{l}(x_3-u)/\mathrm{d}\\ (u-x_1)/d\end{array}\hfill & \begin{array}{l}{x}_2\le u<x_3\\ x_1\le u<x_2\end{array}\hfill \\ 0\hfill & Others\hfill \end{array}$$

$$R_{ij3}=\{\begin{array}{ll}\begin{array}{l}(x_2-u)/\mathrm{d}\\ (u-x_0)/d\end{array}\hfill & \begin{array}{l}{x}_1\le u<x_2\\ x_0\le u<x_1\end{array}\hfill \\ 0\hfill & Others\hfill \end{array}, R_{ij4}=\{\begin{array}{ll}\begin{array}{l}0\\ (u-x_2)/d\end{array}\hfill & \begin{array}{l}u\ge {\mathrm{x}}_1\\ \mathrm{a}\le u<x_1\end{array}\hfill \\ 1\hfill & u<a\hfill \end{array}$$

4.4.2. Membership Value of Negative Indicators

The negative indicators are those that will contribute negatively to the overall goal when their values increase. Their models are:

$$R_{ij1}=\{\begin{array}{ll}\begin{array}{l}1\\ (x_1-u)/d\end{array}\hfill & \begin{array}{l}u<a\\ a\le u<x_1\end{array}\hfill \\ 0\hfill & u\ge x_1\hfill \end{array}, R_{ij2}=\{\begin{array}{ll}\begin{array}{l}(u-x_0)/\mathrm{d}\\ (x_2-u)/d\end{array}\hfill & \begin{array}{l}{x}_0\le u<x_1\\ x_1\le u<x_2\end{array}\hfill \\ 0\hfill & Others\hfill \end{array}$$

$$R_{ij3}=\{\begin{array}{ll}\begin{array}{l}(u-x_1)/\mathrm{d}\\ (x_3-u)/d\end{array}\hfill & \begin{array}{l}{x}_1\le u<x_2\\ x_2\le u<x_3\end{array}\hfill \\ 0\hfill & Others\hfill \end{array}, R_{ij4}=\{\begin{array}{ll}\begin{array}{l}0\\ (u-x_2)/d\end{array}\hfill & \begin{array}{l}u<{\mathrm{x}}_2\\ {\mathrm{x}}_2\le u<b\end{array}\hfill \\ 1\hfill & u\ge b\hfill \end{array}$$

in which $x_0=a,x_3=b,$$x_1$ and $x_2$ equally divide the range of (a, b) into three segments, and a and b are the best and worst indifference point of the indicators.

4.5. Overall Evaluation Result

The overall evaluation result can be obtained through the compounding operation between the fuzzy matrix R and the weight vector A. Specifically, $B=A\infty R$, where $\infty$ is the fuzzy compounding operator using the method of the simple weighted average $M(\cdot ,+)$.

4.6. Analysis of Overall Fuzzy Evaluation Result

After assigning specific values to ${\mathit{v}}_1$, ${\mathit{v}}_2$, ${\mathit{v}}_3$, ${\mathit{v}}_4$, we can then synthesize the vector B based on the weighted average principle.

5. Numerical Example

Having presented the KM performance evaluation framework, we finally demonstrate a numerical example to illustrate the entire process of applying our proposed KM performance evaluation framework in organizations.

We consider a company that uses our proposed model to evaluate its KM performance from last year. In particular, the company establishes two levels of factor sets, creates comment sets, obtains weights with the AHP method, and determines membership values based on the membership functions. Using the learning and growth aspect as an example, the membership function is

$$R4=\left[\begin{array}{cccc}0.72& 0.41& 0.2& 0\\ 0& 0.1& 0.67& 0.24\\ 0& 0.14& 0.78& 0\\ 0.54& 0.43& 0.2& 0\end{array}\right]$$

and the weights are $A4=(0.32,0.22,0.16,0.30)$, so the fuzzy evaluation result is

$$A4\infty R4=\left[\begin{array}{cccc}0.32& 0.22& 0.16& 0.30\end{array}\right]\left[\begin{array}{cccc}0.72& 0.41& 0.2& 0\\ 0& 0.1& 0.67& 0.24\\ 0& 0.14& 0.78& 0\\ 0.54& 0.43& 0.2& 0\end{array}\right]=\left[\begin{array}{cccc}0.3924& 0.3046& 0.4632& 0.0528\end{array}\right]$$

which is $\left[\begin{array}{cccc}0.323& 0.251& 0.382& 0.044\end{array}\right]$ after normalization. Similarly, the fuzzy evaluation results of the other three aspects can be obtained and the overall evaluation result is $\left[\begin{array}{cccc}0.269& 0.260& 0.365& 0.106\end{array}\right]$. If we assign scores to ${\mathit{v}}_1$, ${\mathit{v}}_2$, ${\mathit{v}}_3$, ${\mathit{v}}_4$ as 90, 75, 60, 45, the overall score will be

$$\left[\begin{array}{cccc}0.269& 0.260& 0.365& 0.106\end{array}\right]\left[\begin{array}{c}90\\ 75\\ 60\\ 45\end{array}\right]=70.37$$

which suggests that the evaluation result is above normal and below good, aligned with the current status of the company’s KM initiatives. In particular, the evaluation result indicates that this company focused more on the application side of KM, but did not pay too much attention to knowledge sharing and cultural development, so the company needs to further streamline its business and KM processes.

6. Conclusions

Organizations are constantly seeking innovative ways to exploit the benefits of their knowledge assets. A KM performance evaluation method that can effectively measure the ways to deploy knowledge assets plays a fundamental role in the entire process of KM in organizations. This study proposes a framework for evaluating organizational KM performance through integrating BSC and fuzzy evaluation methods. Specifically, we make a contribution to the current literature along the following lines.

First, we justify the applicability of BSC theory in evaluating KM performance, link specific KM processes with BSC outcomes, and show the causal relationship between KM and BSC. Specifically, we match the finance aspect of BSC with knowledge application, the client aspect with knowledge application and creation, the internal process aspect with knowledge application, acquisition, storage, sharing, and transfer, and the learning and growth aspect with knowledge application and creation.

Second, we create a formal framework to measure the KM performance. In this framework, we demonstrate the indicators to measure each BSC aspect and the justifications of our selection.

Finally, we develop an evaluation model by quantifying some qualitative performance indicators. Through incorporating and synthesizing fuzzy evaluation methods, we show how the indicators in our framework can be quantitatively measured and thus applied in deriving the final evaluation result. We also demonstrate the evaluation process through a numerical example.

Our research provides valuable insights for practitioners to effectively manage knowledge assets in organizations. The proposed approach can be readily implemented in organizations by the manager (e.g., chief information/knowledge officer) to evaluate the performance of KM initiatives through examining the four BSC aspects with our suggested indicators. Future research may focus on the evaluation model by conducting empirical studies to further verify the robustness and applicability of our proposed evaluation framework.