How to Improve Performance and Diversity of Government-Funded Research Institute Ecosystem? Focus on Result Sharing and Feedback Policy

Abstract

Despite its importance to the performance outcome of an organization, there are very few studies on how feedback mechanism impacts ecosystems of government-funded research institutes (GIs). This study focuses on the effect of the feedback mechanism on the average performance and diversity of a GI ecosystem. Feedback mechanisms consisted of feedback strategy and degree of result sharing. An agent-based model that embeds a genetic algorithm to replicate a real GI ecosystem was used. It was found that relational patterns between average performance and degree of result sharing varied by type of feedback policy. In contrast, convergence time, which refers to the average period of settling the stable state in the perspective of ecosystem diversity, depends on the ratio of result openness rather than the type of feedback policy. This study suggests two plans to improve the GI assessment system by changing the degree of result sharing and feedback type.

1. Introduction

The government-funded research institute (henceforth GI) is one of the main subjects of the national innovation system and is established and operated to promote the development of the national economy by innovating national science and technology and strengthening national competitiveness [1,2]. Therefore, it is important to improve the efficiency and research performance of national research and development (R&D) investment by efficiently managing and utilizing the research performance of GIs through an optimal assessment system [3,4,5]. An attractive assessment system has been known to be an efficient way to enhance organizational performance [6,7]. However, even if there is a clear evaluation metric, it cannot always guarantee good effects [8]. The influence of feedback systems may increase when the gap between organizational goals and status quo is widened [9]. However, feedback systems cannot always guarantee good effects [8] because of antecedents such as frequency [10,11,12,13], intensity [10], and individual context [14]. Indeed, the assessment system of GIs in Korea is not an exception.

In the 1990s, the Korean government established an assessment system for GIs to improve their outcomes. Concerning this evaluation system, studies suggested several guidelines for upgrading the GI assessment system. Most suggestions emphasized only internal factors [15,16,17,18,19,20], such as validity of criteria, evaluator’s ability, and reliability of the assessment system itself, rather than the feedback mechanism. However, the feedback method of assessment should be considered because feedback is critical to improving the performance of an organization [21,22].

The present study focuses on the effect of the type of feedback policy, which has seldom been explored. This study aims to examine two aspects: (1) the effect of changing the incentive policy and (2) the degree of the result disclosure on average performance and diversity of the ecosystem of GIs. Based on the findings, this study will provide guidelines for strategies to enhance the efficiency of GIs using feedback mechanisms including feedback policy and degree of sharing assessment results.

In Section 2, previous studies of the feedback system are presented to explain the effect of the feedback system and criticism of existing feedback systems of Korea. We revisited feedback research investigating the factors which improve the impact of feedback. Furthermore, this chapter provides criticism that supports the lack of ex-post utilization of feedback results. In Section 3, the methodology of this study is introduced. This study adopted an agent based model simulation based on organizational learning. In the present study, three layers, including agent, system, and ecosystem are assumed for describing a bottom-up process. Five agent-based model (ABM) experiments are conducted by the type of feedback policy based on assessment result sharing. From this experiment, we investigate the relational patterns between the average performance and degree of result sharing determining the performance and diversity of GI’s ecosystem. In Section 4, details of the experiments’ results are given. The result of the experiments implies two ways to improve the performance of the ecosystem. Lastly, we discuss how to interpret and apply our findings to a real-world viewpoint.

2. Literature Review

2.1. GI Assessment System and Its Critics

Organizational assessment plays a significant role in improving the quality of the organization through a self-sustaining feedback process by contributing to the organization’s goal setting and creating a competitive environment [6,7,23]. As GIs have a unique ownership structure and behavioral characteristics, their assessment and evaluation should consider inherent inefficiency, fair compensation, capacity of development of organizational members, and organizational efficiency [23].

The Korean GI assessment system was established in 1991. Since this system includes assessment and proportional response toward GIs, it assumes the role of a feedback system. Unlike private firms, GIs have focused on the science and technology of public areas that are difficult for universities and firms to access [24]. It is thus important to consider their roles and characteristics when they are evaluated. Figure 1 presents a flow chart of the feedback system for GIs in Korea. The Korean government-organized GI assessment system has three performers who assume unique responsibilities. Each actor plays the role of self-assessment, meta-evaluation, and final decision making.

Since the inception GI assessment, there has been much criticism of the effectiveness of the public institute assessment system. Critical studies mainly relate to determinants of acceptance towards assessment results, such as reliability, validity, and the utilization of results [25]. There are two major streams of critiques observed in previous studies.

The first criticism refers to problems of the assessment system, such as lack of assessment capability, inconsistent criterion, and structural faults of the operation system [16,25,26,27]. In particular, criteria of assessment have been considered critical factors in previous studies. The problem lies in the fact that various criteria do not reflect actual heterogeneous properties of GIs. To establish solutions, previous studies proposed frequent changes of the assessment metric [27], reviewing insufficient measurement of performances [28], uniformity of final assessment [26] and role of public R&D management agencies [29] as plausible causes of the problem. In addition, selecting an appropriate assessment target [16,26], size effect bias [30], networking [31], gaps between goals of assessment systems and institutes [16], and acceptability of results [25] may be considered factors for an efficient institute assessment system.

The second criticism is related to ex-post utilization of assessment results. Hwang and Kang (2004) [27] and Son (2007) [28] argued that lack of reliability of assessment results could cause unacceptability of the assessment system. Further, insufficient follow-ups [27] and lack of detailed information of feedback [16,26,27,28] could be critical issues in enhancing assessment efficiency. In addition, selecting an appropriate assessment target [16,26], size effect bias [30], networking [31], gaps between goals of assessment systems and institutes [16], and acceptability of results [25] may be considered factors for an efficient institute assessment system.

2.2. Organizational Feedback and Its Effect

Previous organizational assessment studies have focused on methods to enhance the performance and effectiveness of an organization. According to them, the feedback system and a meta-evaluation have been considered, as critical factors for improving the organizational performance and capability feedback system are based on the learning behavior of each member [12,32,33]. Hence, most studies agree with the result that the feedback system may increase organizational performance regardless of its type [21,22]. However, the effect of the feedback system can be changed by the properties of the system itself [21,22,34] or the contextual condition of the organization [35,36,37,38,39]. The factors suggested in the previous studies are distinguished into two categories.

Table 1. Two category of factors determining the effect of feedback.

Category	Content	Reference
Ex-ante factor	Defensive deposition decreases feedback performance.	[40]
	Public self-consciousness increases the needs of feedback	[41]
	Individualism increases feedback performance	[42,43,44]
	Difficult goal increases feedback performance	[45,46,47,48]
	Gap between goal and status of quo increases feedback performance	[9,49]
	Large scale of goal increases feedback performance	[50]
	Frequent assessment increase feedback performance	[21,22,34,51]
	Frequent assessment decrease feedback performance	[52,53]
	Reliability increases feedback performance	[12]
Ex-post factor	Negative feedback system is more effective than positive one	[54,55,56,57,58,59,60]
	Positive feedback system is more effective than negative one	[22,39,48,61,62]
	Positive delivery method increases acceptability of feedback result	[38]
	High result sharing decreases the needs of feedback	[14]
	Detail feedback increases short-term feedback performance	[10,63]

shows the classification of factors affecting the feedback performance. The ex-ante factor is defined as an antecedent of feedback effectiveness before the assessment result is released. Contrary to that, the ex-post factor refers to the action utilizing the assessment result. Despite the previous studies that emphasized both sides of the feedback system, ex-ante and ex-post, the GI feedback system in Korea was likely to consider the ex-ante factors. In particular, the reliability of assessment process was the most important point of the system.

Previous studies relatively emphasized the ex-ante aspect of the feedback system. Despite the various ex-post factors presented in feedback theory researches (see Table 1), only the feedback detail was included in the discussion about the GI feedback system in Korea. Furthermore, in the theoretical approach of the feedback system, it was hard to find a research investigating multiple factors simultaneously. The present study has advantages in those points. This study not only focuses on the ex-post factors but also considers two aspects, these are feedback policy and the degree of result sharing at the same time.

3. Research Model

As mentioned earlier, the existing feedback system does not seriously discuss the ex-post perspective and multilateral relationships. The present study adopts an ABM simulation method to understand the feedback mechanism of the GI ecosystem. In our ABM simulation, ex-post factors of the feedback system involve result sharing activity and a type of feedback policy. Result sharing is an interactive activity that transforms a feedback system into a complex adaptive system [64].

GI’s feedback system is a complex adaptive system composited by non-linear causalities and emergent phenomenon [65]. Hence, this study built an ABM to examine the impact of the result sharing and the feedback policy on the evolution of the GI ecosystem on the performance and diversity perspectives. Therefore, constructing an ABM in this study is an effective way to describe social phenomena caused by interaction among the agents [66]. In this ABM, microscopic interactions, such as result sharing, are based on the evolutionary process using the genetic algorithm [67].

3.1. Agent Based Model Simulation (ABMS)

Unlike analytic methodologies, the ABM can interpret social phenomenon based on a bottom-up process. The ABM has the advantage of dealing with unstructured problems, such as ambiguous assumptions, and phenomena controlled by individual levels through simplifying complex systems [68]. Thus, ABM has been used to analyze complex and large sizes of the system through interactions of autonomous entities [69]. In the ABM simulation, since a result is not a sum of parts, the ABM simulation should be modularized into multiple layers [70]. This study assumed three layers consisting of “agent later”, “system layer,” and “ecosystem layer”. Figure 2 presents a detailed description of the ABM simulation research model. “Agent layer” refers to a microscopic level of an ecosystem and includes multiple research institutes, which is the minimum unit of the GI ecosystem. Based on the bottom-up perspective, all decisions are conducted in this layer. “System layer” defines the interaction between the agent layer and the ecosystem layer. Although no autonomous agent exists in this layer, the system layer may express how to interact among agents in given conditions. The “ecosystem layer” provides a macroscopic view to observe emergent phenomena that are being investigated. Similar to the system layer, the ecosystem layer does not have actual agents. However, this layer relays information on the current environment. This information may influence the decisions of each agent. Furthermore, this layer shows macroscopic patterns, such as average performance, diversity, or regular tendency. While each layer can manipulate its own layer, interactions between layers are possible.

Within the basic framework of the genetic algorithm (GA), various approaches depend on the method of recombination and sampling of the mating pool. We used alternative strategies because of two limitations in the basic process. First, in the biological context, it is very natural that only two entities join in the mating process. However, more than two entities can join the mating process in a social science context. Second, there is a deceptive problem. Strong searching power, which covers the entire sample space, may be effective for fast and short-term evolution. However, it not only generates a premature solution but also contradicts the assumption of bounded rationality. To solve these problems, we used a sampling ratio (β) representing the characteristic of bounded rationality to prevent the pressure vitiating the performance of the algorithm [71].

3.2. Model Layers

3.2.1. Agent Layer

The agent layer refers to the institute layer to which GIs belong. The agent layer consists of heterogeneous and autonomous entities who interact with other entities to maximize their objective function [70]. In this layer, the initial attributes of each institute are determined. A set of properties of institute i is described as a string of genes ($S_i$). Each property is expressed by an integer from 0 to 2. Performance (p), learning rate (π), forgetting rate ($\mathsf{\lambda }$) and self-adjustment rate ($\mathsf{\gamma }$) are assigned randomly. The organizational performance, learning rate, and forgetting rate are based on Levitt and March (1988) [55], and March (1991) [56]. Individual performance is calculated through the concept of fitness in the evolutionary process.

$$P_i=f\left(S_{i,t},S_t^{ideal}{S}_{opt,t}\right)=\frac{1}{2n}{{\displaystyle \sum }}_{j=1}^n(2-|{\delta }_{i,j}-{\delta }_j^{ideal}|)$$

(1)

$$\begin{gathered} n=\mathrm{number}\mathrm{of}\mathrm{properties} \\ {\delta }_j^{ideal}=j\mathrm{th}\mathrm{ideal}\mathrm{attribute} \\ {\delta }_{i,j}=j\mathrm{th}\mathrm{attribute}\mathrm{of}\mathrm{institute}i \end{gathered}$$

Self-adjustment rate “γ” is a unique attribute of this study. This attribute refers to the action of a spontaneous effort for improvement without any external interference. This study assumes that self-adjustment is determined by the performance observed just before period.

Table 2. The list of attribute in the agent layer.

Variable.	Notation	Description
Institute id	i	i ∈ {1, 2, …., N}
Attribute string	$S_i$	$S_i=\{a_{0,i}$, $a_{1,i}$, $a_{2,i}$ …., $a_{n,i}$} $a_i\in$ {0, 1, 2}
Performance	$P_i$	$P_i=f$($a_{0,i}$, $a_{1,i}$ … $a_{n,i},S_t^{ideal})$ $S_t^{ideal} is an ideal condition of attribute string$
Learning rate	${\pi }_i$	$0\le {\pi }_i\le 1.0,$${\pi }_i$—uniform distribution
Forgetting rate	${\lambda }_i$	$0\le {\lambda }_i\le 0.1,$${\lambda }_i$—uniform distribution
Self-adjustment rate	${\gamma }_i$	$0\le {\gamma }_{i,0}\le 0.1$, ${\gamma }_{i,t}$ = $f\left(P_{i,t-1}\right)$

shows details of each attribute in the agent layers.

3.2.2. System Layer

The system layer manages a middle level of the ecosystem providing predefined rules of The system layer manages a middle level of the ecosystem providing predefined rules of interactions. In this layer, interaction among the institutes are presumed by an evolutionary process known as the genetic algorithm as suggested by previous studies. The GA describes an evolutionary process through mutual learning, which has special mating rules among agents. The evolutionary process based on GA has been utilized in a social context such as inter-organizational learning [72] and intra-organizational learning [73,74]. At this stage, the higher the performance of the institute $i$, the more likely it is to be chosen as a learning object. The imitation probability is the same as the learning rate (${\pi }_i$).

A strong searching power, which covers the entire sample space, not only generates a premature solution but also violates the assumption of bounded rationality. To solve these problems, we used a sampling ratio (β) representing the characteristics of bounded rationality to prevent the pressure vitiating the performance of the algorithm [71]. The sampling ratio β is to control the selection pressure of the GI ecosystem. In general, the network of private firms has a central node that is incumbent or most benefitted, whereas universities and public research institutes do not have nodes that play central roles in their ecosystems [75,76]. Hence, sampling ratio β is defined as a probability derived from a uniform distribution. When β is 1.0, the model considers the entire population as a sampling pool and generates the highest selection pressure, and as β closes to zero, the evolution of the system becomes a random walk process. In addition, the total number of institutes (N), time (t), and the amount of incentive (v) are defined in this layer (see

Table 3. The list of attributes in the system layer.

Variable	Notation	Description
Population	N	number of institutes in the institute layer
Incentive	v	amount of incentive to high performance group
Time	t	t ∈ {1, 2, …., max iteration}
Sampling ratio	${\beta }_i$	$0\le {\beta }_i\le 1.0$, ${\beta }_i$ = ${\beta }_{j}wheni\ne j$

).

3.2.3. Ecosystem Layer

Generally, an emergent phenomenon is likely to occur at a different hierarchy [77]. The ecosystem layer has no authority to control the parameter but plays a role as a window for observing the macroscopic phenomenon. Through this layer, we can observe the average performance and diversity of GI ecosystem. The average performance of the ecosystem refers to the overall fitness toward the environmental properties. To calculate the performance, the ecosystem layer provides an ideal status of attributes. Similar to the attribute string of each institute, ideal attributes are presented as a string. The ideal string at period t $\left(S_t^{ideal}\right)$ is dependent on both the initial condition and internal assessment. Average performance is calculated as shown below.

$$\begin{gathered} P_{avg,t}=\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{P}_i=\frac{1}{2Nn}{{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^n(2-|{\delta }_{i,j}-{\delta }_{opt,j}|) \\ N=\mathrm{number}\mathrm{of}\mathrm{population}\mathrm{in}\mathrm{the}\mathrm{ecosystem} \end{gathered}$$

(2)

Internal assessment is a unique characteristic of the GI feedback system in Korea. When performance and diversity are calculated, the feedback system reflects the metric of internal assessment organized by each institute. Internal assessment can be included in the initial criteria when it is approved by upper organizations. In this study, the reflection of internal assessment (c) is defined as a probability between 0 and 0.1. The list of attributes used in the ecosystem layer are shown in

Table 4. The list of attribute in the ecosystem layer.

Variable	Notation	Description
Ideal attribute string	$S_t^{ideal}$	$S_i=\{a_{0,1}$, $a_{1,1}$, $a_{2,1}$ …., $a_{n,1}$} $a_i\in$ {0, 1, 2}
Reflection ratio	c	$0\le c\le 0.1$
Average performance	$P_{AVG}$	$\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{p}_i$
Diversity	$D_t$	Hamming distance at period t

.

Because of its importance, maintaining diversity has been investigated in previous studies [78]. There are four representative ways to measure ecosystem diversity as shown in

Table 5. Measuring the diversity.

Hamming Distance Between Vector X and Y: $\mathbf{D}\left(\mathbf{X},\mathbf{Y}\right)$	Euclidian Distance Between Vector X and Y: $\mathbf{D}\left(\mathbf{X},\mathbf{Y}\right)$
$I={{\displaystyle \sum }}_{j=0}^L{I}_j, I_j=\{\begin{array}{c}0 if x_j=y_j\\ 1 if x_j\ne y_j\end{array}$ $\mathrm{D}\left(\mathrm{X},\mathrm{Y}\right)=\frac{I}{L}$	$\mathrm{D}\left(\mathrm{X},\mathrm{Y}\right)=\sqrt{{\displaystyle {\displaystyle \sum }_{i=0}^N}{(x_i-y_i)}^2}$
Connection matrix: D(X,Y)	Information entropy: PD
S(X,Y) = ${{\displaystyle \sum }}_{i,j}\frac{(x_{ij}|x_{ij}=y_{ij}=1)}{n}$ D(X,Y) = 1 − S(X,Y)	$H_i=-{{\displaystyle \sum }}_{c\in C}p{r}_{ic}ln(p{r}_{ic}),\mathrm{where}p{r}_{ic}=\frac{n{a}_{ic}}{N}$ $\mathrm{n}{a}_{ic}:\mathrm{number}\mathrm{of}\mathrm{appearance}\mathrm{of}c\mathrm{at}\mathrm{locus}i$ $\mathrm{C}:\mathrm{number}\mathrm{of}\mathrm{cities}\mathrm{should}\mathrm{be}\mathrm{visited}$ $\mathrm{PD}=\frac{{\displaystyle \sum }D\left(X,Y\right)}{N}$

. In this study, the dimension of the attribute string of institutes is homogeneous. In addition, individual attributes are expressed by the form of a nominal variable. The Euclidian distance and connection matrix-based diversity are more appropriate to geometrical aspects. Information entropy is relevant to stochastic uncertainty. Thus, it is natural to adopt the “Hamming distance” as a way to measure ecosystem diversity.

3.3. Experiment

In this section, experiments based on ABM simulation are introduced. As this study assumed in a previous section, both the feedback policy and the degree of result sharing are the ex-post strategies of a feedback system. However, since the degree of result sharing is a continuous number between 0 and 1, it is hard to design the experiment model in that manner. Hence, this study organizes five experiments based on the type of feedback policy rather than the result sharing. To enhance the validity, all experiments were conducted 100 times and the results were derived from the average of repeated results.

According to previous research, there are two types of feedback: positive and negative feedback. Both types of feedback are known to be effective methods for improving organizational performance [21,22,33,34]. In addition, the target of feedback may be considered another aspect of the type of feedback. In the present research model, types of GI are distinguished by two groups involving high performer and low performer (see

Table 6. ABM experiment design.

Type	Target	Experiment
No feedback	High and low performance group	Experiment 1
Positive feedback	High performance group	Experiment 2
	Low performance group	Experiment 4
Negative feedback	Low performance group	Experiment 3
Positive and negative feedback	High and low performance group	Experiment 5

).

3.3.1. Experiment 1: No Policy

The first type of feedback policy is ‘no policy’, which does not have any feedback. The GIs become autonomous entities that are dependent on their decision making. Under the ‘no policy’ condition, all parameters are invariant because there is no evidence to evaluate. The function of institute i can be shown as:

$$\mathrm{Agent}i=\{S_{i,t},P_i\left(S_{i,t-1}\right),{\beta }_{i,t},{\pi }_{i,t},{\lambda }_{i,t}, {\gamma }_{i,t}\}\mathrm{at}\mathrm{policy}\mathrm{type}=1$$

(3)

$$\begin{gathered} S_{i,t}:\mathrm{String}\mathrm{of}\mathrm{knowledge}\mathrm{of}\mathrm{institute}i\mathrm{at}t; \\ {P}_i\left(S_{i,t-1}\right):\mathrm{Performance}\mathrm{of}\mathrm{institute}i\mathrm{at}t-1; \\ {\beta }_{i,t:} \mathrm{Sampling}\mathrm{ratio}\mathrm{of}\mathrm{institute} i \mathrm{at} t; \\ {\pi }_{i,t}:\mathrm{learning}\mathrm{rate}\mathrm{of}\mathrm{institute} i \mathrm{at} t; \\ {\lambda }_{i,t}: \mathrm{forgetting}\mathrm{rate}\mathrm{of}\mathrm{institute} i\mathrm{at} t; \\ {\gamma }_{i,t}:\mathrm{self}-\mathrm{adjustment}\mathrm{rate}\mathrm{of}\mathrm{institute} i \mathrm{at} t; \end{gathered}$$

3.3.2. Experiment 2: Positive Feedback to the High-Performance Group

The second type is the ‘positive feedback to the high-performance group’ policy. In this model, the positive feedback is only provided to the high-performance group. The high-performance group is filtered by the predefined ratio ($r_{high\_perf})$. The amount of incentive is determined by the incentive parameter ($v$) between from −1.0 to 1.0.

$$\mathrm{Agent}i=\{S_{i,t},P_i\left(S_{i,t-1}\right),{\beta }_{i,t},{\pi }_{i,t},{\lambda }_{i,t}, {\gamma }_{i,t}\left(P_{i,t-1},{\gamma }_{i,t-1}\right)\}$$

(4)

$$\begin{gathered} {\gamma }_{i,t}\left(P_{i,t-1},{\gamma }_{i,t-1}\right),=\{\begin{array}{c}\left(1+v\right)\times {\gamma }_{i,t-1} if P_{i, t-1}>\left(r_{high\_perf}\times \frac{{{\displaystyle \sum }}_{i=0}^N{P}_{i,t-1}}{N}\right)\\ {\gamma }_{i,t} elsewhere\end{array} \\ v:\mathrm{quantity}\mathrm{of}\mathrm{incentive} \\ {r}_{high\_perf}:\mathrm{ratio}\mathrm{of}\mathrm{high}-\mathrm{performance}\mathrm{group} \end{gathered}$$

3.3.3. Experiment 3: Positive Feedback to the Low-Performance Group

Although this policy seems to defy social common sense, it has been used in non-industrial areas, such as an international aid programs, technology, and public education. The definition of a low-performance group follows the experiment 2.

$$\mathrm{Agent}i=\{S_{i,t},P_i\left(S_{i,t-1}\right),{\beta }_{i,t},{\pi }_{i,t},{\lambda }_{i,t}, {\gamma }_{i,t}(P_i|P_i\le r_{low\_perf}\times \mathrm{mean}\left(P_i\right))\}$$

(5)

$$f_t\left(P_{i, t-1}\right)=\{\begin{array}{c}\left(1+v\right)\times {\gamma }_{i,t-1} if P_{i, t-1}\le \left(r_{low\_perf}\times \frac{{{\displaystyle \sum }}_{i=0}^N{P}_{i,t-1}}{N}\right)\\ {\gamma }_{i,t} elsewhere\end{array}$$

3.3.4. Experiment 4: Negative Feedback to the Low-Performance Group

The third type of feedback is “negative feedback to low-performance groups,” wherein low-performance institutes receive penalties. A low-performance group is determined in the same manner as feedback policy 2. Contrary to feedback policy 2, negative feedback can pressure low-performance institutes to change their current statuses. A detailed description of the agent in feedback type 3 is shown as Equations (4) and (5).

$$\mathrm{Agent}i=\{S_{i,t},P_i\left(S_{i,t-1}\right),{\beta }_{i,t},{\pi }_{i,t},{\lambda }_{i,t}, {\gamma }_{i,t}(P_i|P_i\le r_{low\_perf}\times \mathrm{mean}\left(P_i\right))\}$$

(6)

$$f_t\left(P_{i,t-1}\right)=\{\begin{array}{c}\left(1-v\right)\times {\gamma }_{i,t-1} if P_{i,t-1}\le \left(r_{low\_perf}\times \frac{{{\displaystyle \sum }}_{i=0}^N{P}_{i,t-1}}{N}\right)\\ {\gamma }_{i,t} elsewhere\end{array}$$

3.3.5. Experiment 5: Positive Feedback to the High-Performance Group and Negative Feedback to the Low-Performance Group (Mixed Policy)

The last type of feedback policy is a mixed policy of Experiments 2 and 4. This is a similar policy with actual GI system assessment. The low-performance and high-performance entities are designated akin to or in the same manner as previous feedback polices.

$$\mathrm{Agent}i=\{S_{i,t},P_i\left(S_{i,t-1}\right),{\beta }_{i,t},{\pi }_{i,t},{\lambda }_{i,t}, {\gamma }_{i,t}(P_i|P_i\le r_{low\_perf}\times \mathrm{mean}\left(P_i\right))\}$$

(7)

$$f_t\left(P_{i,t-1}\right)=\{\begin{array}{c}\left(1-v\right)\times {\gamma }_{i,t-1} if P_{i,t-1}\le \left(r_{low\_perf}\times \frac{{{\displaystyle \sum }}_{i=0}^N{P}_{i,t-1}}{N}\right)\\ \left(1+v\right)\times {\gamma }_{i,t-1} if P_{i,t-1}>\left(r_{low\_perf}\times \frac{{{\displaystyle \sum }}_{i=0}^N{P}_{i,t-1}}{N}\right)\\ {\gamma }_{i,t} elsewhere\end{array}$$

4. Result

From the simulation experiments, we captured the effect of the degree of result sharing and the type of feedback policy toward the average performance and diversity.

4.1. Average Performance

We captured relations between average performance and type of feedback policy with the changing of the degree of result sharing. According to results presented in

Table 7. Summation of the simulation results.

Model.	Experiment 1	Experiment 2	Experiment 3	Experiment 4	Experiment 5
	No Feedback	Positive Feedback to High-Performer	Positive Feedback To Low-Performer	Negative Feedback to Low Performer	Mixed Policy
Average Performance

Note: horizontal-axis refers to “the degree of result sharing” and vertical-axis refers to “the average performance of ecosystem”.

, average performance of the ecosystem represents different patterns of feedback policy.

4.1.1. Experiment 1: No Policy

Under the “no feedback policy,” the average performance monotonically increases with the degree of result sharing. The reason is simple. If there is no evidence to choose a target organization, the learning behavior only depends on the sample size that can be actually used. However, unlike learning behavior involving new costs, situations without feedback do not bring any benefits to the institute. Thus, this result can be observed when institutes have other factors stimulating self-adjustment such as human resources [79] and culture [80], except feedback from the evaluation system.

4.1.2. Experiment 2 and 3: Positive Feedback Policy

Both, the “positive feedback to high performance” model and “positive feedback to low performance” models show a rotated S-shape pattern similar to the sine graph. As positive feedback can be a catalyst in stimulating improvement in the ecosystem’s performance, each institute tries to learn from institutes that received better evaluations. In this context, providing assessment information increases the possibility that other institutions choose better institutions. Likewise, the higher the degree of sharing of evaluation results, the greater the size of samples available, and the more likely it is to select organizations with high performance institutes. However, since the perfect information of the assessment result saturates the GI ecosystem in the aspect of evolution, imperfect information is more effective to improve average performance in the long term. [74]. According to the result of our experiment, optimal sharing proportion is between 0.2 and 0.5. Relations between the sharing proportion and average performance show the same patterns regardless of performance of subjects who received positive feedback. In other words, positive feedback policy provides a rotated S-shaped pattern between sharing level of assessment and mean performance, once provided with positive feedback, regardless of whether grantees received excellent performances. Consequently, two conclusions from results of models which only embedded the positive feedback policy were found. One is a rotated S-shaped pattern between the degree of result sharing and average performance of GI ecosystem, and the other is a type of feedback that is more important than the excellence of the grantee.

4.1.3. Experiment 4 and 5: Negative Feedback Policy

From Experiments 4 and 5, we captured the inverted U-shaped pattern. When negative feedback exists, an inverted U-shape relationship can occur between average performance and sharing level. This pattern of relationship refers to the result sharing proportion maximizing the average performance of the ecosystem that is located between 0 and 1. Therefore, we suggest that a moderate sharing level is more effective than an extreme sharing level when a “negative feedback policy” is adopted. In addition, this experiment provides evidence on the effect of simultaneous use of positive and negative feedback. The result shows that the effect of a “negative feedback policy” is more dominant than that of “positive feedback” as the same pattern emerges with the results of the “negative feedback policy” (Experiment 4).

4.2. Convergence Time

Unlike results of the average performance of the ecosystem, convergence time of the GI ecosystem only relies on the degree of assessment result sharing. This result is summarized in

Table 8. Summation of the simulation results.

Model	Experiment 1	Experiment 2	Experiment 3	Experiment 4	Experiment 5
	No Feedback	Positive Feedback to High-Performer	Positive Feedback to Low-Performer	Negative Feedback to Low Performer	Mixed Policy
Convergence speed

Note: horizontal-axis refers to “the time(period)” and vertical-axis refers to “the convergence time.

. Indeed, a decrease in diversity of an evolutionary system is an inevitable phenomenon without powerful external interventions. The convergence time, which indicates time taken for diversity to converge, is rapidly shortened as the degree of sharing increases.

There are two interpretations of shorter convergence time. Firstly, diversity of the GI ecosystem has a higher probability to not be undermined. Maintaining diversity of ecosystems guarantees stability of the system from unexpected changes in the external environment as well as loss of current knowledge. Preservation of system diversity also increases likelihood of emerging new knowledge in the ecosystem, which may lead to improved decisions. GI ecosystem in experiments maintains quantitative diversity in the very low result sharing level According to Figure A2 in Appendix A, the diversity of GI ecosystem is highest when the degree of result disclosure (β) is 0. If the rate of openness is under 0.2, diversity begins to decrease sharply and shows similar levels of quantitative diversity in the rest of the domain. Results of convergence time show that it is hard to maintain high diversity of GI ecosystem except in the range with low degrees of result sharing. However, the loss of diversity when seeking optimum solutions is natural, so the present study emphasizes another aspect to interpret short convergence times. The second implication of the experiment result refers to the possibility of an efficient learning process. The efficiency of organizational learning is a crucial factor because it is difficult to repeat enough organizational learning processes in the actual GI ecosystem, such as in the current model of research. The short convergence time in this study model means that the GI ecosystem reaches its optimal state quickly. Figure 3 shows a relatively short convergence time when the degree of result sharing is higher than 0.2 (β > 0.2). Therefore, the GI ecosystem is efficient at the most openness proportion (β > 0.2), and the inefficiency of the learning process is observed only at a low degree of assessment result sharing.

5. Discussion and Conclusions

It was previously rare to develop policies that considered the influence of feedback type and sharing level in regard to assessment results. Likewise, assessment systems of GIs in Korea have seldom considered systematic or multi-dimensional perspectives for the effect of the result sharing level and type of the feedback policy. Hence, this study aims to determine the effect of changing feedback policy and the degree of result sharing on the average performance and diversity of GI ecosystems. Ultimately, several practical strategies are suggested based on results of ABM experiments. Based on results of five experiments, this study proposes a practical method to enhance the effectiveness of the assessment system of organizations with similar goals in terms of feedback type and sharing levels of assessment results.

According to experiments in this study, switching types of feedback policy transform unique patterns between degrees of disclosing assessment results and average performance of GI ecosystems. With the exception of Experiment 1 (no policy), the middle level of result sharing is likely to maximize average performances rather than extreme levels. Relatively, positive feedback is likely to decrease optimum levels of result sharing as compared to negative feedback. The result of this study shows that there is no absolutely dominant strategy. This is a different argument compared to previous studies that emphasized the effect of negative feedback [54,56,57,60].

When an organization adopts positive feedback policy, two types of strategies can be considered. The first strategy shares complete results of assessments to all participants and can provide transparency of the assessment system, mutual trust, and reliability towards the GI assessment system. Due to this advantage, a broader range of learning objectives are acquired, and the probability of discovering optimal target institutes in a specific time period increases. Contrary to this method, another strategy uses a low ratio of result sharing. Indeed, fluctuation in the environment requires a new capability that was not crucial in the past. However, the current metric of the GI assessment system may not include all factors, which must be evaluated. From the perspective of handling uncertainty of future environments, providing only a part of the sample is necessary to preserve the potentially beneficial attributes that each GI possesses.

The GI assessment system in Korea has applied both negative and positive incentive policies utilizing fully opened assessment results. As aforementioned, the Korean GI assessment system can be improved by changing either the type of feedback policy or the degree of result openness. At first, since the effect of negative incentive is dominant, mixed feedback policy shows a similar trend to the negative one as seen in the previous experiment 3. If the government only focuses on the aspect of the performance, it is effective to change the current policy based on the result of Experiment 5 in the present study. However, transparency should be handled carefully because it determines external acceptance of the GI assessment system. Consequently, this strategic decision making entails a dilemma between transparency and effectiveness. When the government decreases sharing rates of assessment results to maximize performance, the monitoring function of assessment systems will be weakened. In other words, smooth acceptance of policies is threatened by lack of transparency. However, another strategy is possible to maintain the sharing level and to choose “positive feedback to high performance” feedback type. Although this is not the best solution in the context of efficiency, transparency toward GIs can thus be acquired. From the perspective of GIs, this alternative can resolve cognitive threat of unfair disclosure of results. This strategy is effective in improving social trust and acceptance of the evaluation system, jeopardizing some improvements in organizational performance. However, absence of negative feedback may not only abate the power of correction and tangible effectiveness but also lead to inefficient exploration of an optimal solution. Of course, we can consider the “negative feedback to low-performance group” policy as another solution, but this type of policy is realistically difficult. If new feedback or self-motivation are not provided, each GI makes an effort to lower its own performance to minimize costs.

Table 9. Comparison between different improvement strategy of the GI assessment system.

Strategy	Changing The degree of Result Sharing	Changing the Type of Feedback Policy
	Perfect Sharing → Low Sharing	Mixed Policy → Positive feedback to High-Performance Group
Goal	Mid ~ Long-term objectives	Short ~ Mid-term objectives
Benefit	- Maximizing effectiveness of assessment system on the aspect of both performance and diversity - Relatively strong correction towards individual and organizational behavior	- Relatively improving effectiveness of the assessment system on the aspect of both performance and diversity - Keeping public transparency and trust toward the GI assessment system - Fair sharing of the assessment results
Cost	- Declining transparency and trust towards the GI assessment system - Unfair selection of institute	- Relative weak correction power toward organizational and individual behavior - Inefficiency of the optimal solution exploration

describes the summarization of these strategies.

Despite theoretical contributions of this study, empirical evidence of the feedback theory is required. In fact, experiments based on computer simulation have an advantage on prediction of social phenomenon and analysis on the basis of complex adaptive systems [81,82]. However, simulation methodology is not suitable to verify and generalize existing theories. This study has a similar limitation. Future studies should focus on ex-post analysis, such as real-world case studies or empirical analyses. Further practical research will provide more implementable solutions. It is thus important to figure out the exact point of result sharing ratio to maximize objective functions such as sales, patents, knowledge, and products.