The North Korean Economy: Sustainable or Muddling-Through?

Confronting the continuing economic sanctions, the flood disaster, and the COVID 19, North Korea is now in a critical phase whether the North Korean economy will shift to a sustainable economy or, as it has done in the past three decades, barely manage to maintain at a survival level. To explore the question, this study investigates North Korea’s economic policy changes by extracting the keywords from the North Korean economic journal Economic Research, which is a representative state published document. We use a text mining approach to overcome limitations, such as the lack of objectivity and accuracy of the classification of topics in manual text analysis for North Korean studies. Contrary to the conventional assumption that the North Korean economic policies are driven by dogmatic Juche (self-reliance) ideology, we find that the strategy of economic policies differs depending on the leader and political and economic circumstances at the time. In particular, since Kim Jong-un assumed office in 2011, the topics have diversified, and now cover a wide range of subjects on capitalism. Nevertheless, without North Korea’s decisive reform measures and international cooperation, it will be hard to observe that the North Korean economy can go beyond ‘muddling-through’ for sustainable development.


Introduction
North Korea's leader Kim Jong-un held the 5th plenary session of the 7th Central Committee in December 2019 and presented a 'new path' that calls for self-reliance on the economy, political and military stability, and party control as key elements. However, at the plenary session of the 7th Party Central Committee, which was held only half a year later, Kim Jon-un officially recognized the 'poor economic performance' and to 'adjust and change' some of the policy tasks decided at the 5th meeting, and to hold another evaluation meeting on economic performance at the 8th Central Committee meeting in January 2021. The statement of the party's plenary session pointed out that "(the) internal and external situation has continued and unexpected challenges have failed to improve the economic situation, failed to meet the planned growth target of the national economy, and failed to improve the lives of residents." As Kim Jong-un himself mentioned, the internal moves in the North confirm the challenges faced by the North Korean authorities in triple trouble: The continuing economic sanctions, the flood disaster, and the COVID19. It is an important turning point that shows whether the North Korean economy will shift beyond recovery to a sustainable economy or, as it has done in the past three decades, barely manage to maintain at a minimum level of survival.
To be sure, the North's economic difficulties are not new, and they are the result of the accumulation of structural problems accumulated over the past decades. Both the death of Kim Il-sung in 1994 and

Analysis of North Korean Literature
In this study, the title and keywords of a paper in Economic Research is an analytical unit that can track key agenda and policy changes that North Korea has emphasized for its economy. In particular, this research investigates whether any changes in economic policies introduced and implemented by Kim Il-sung, Kim Jong-il, and Kim Jong-un can be classified by the characteristics of the keywords used in the titles at the time of their respective leadership. This investigation expects to help us comprehend North Korea's strategic directions, as well as its challenging issues for the country.
Artificial intelligence text mining and machine learning, which can analyze meaningful results from a limited number of primary data, have been actively utilized recently. However, North Korea-related research and national statistics are severely limited in terms of accessibility and credibility, thus, being the biggest obstacle in this study. The main significance of this study is that, despite the limited data, results can be found with advanced text mining techniques. Additionally, the significance of this study is clear in the comparative study of different governance systems. Big data that is composed of text mining can also be analyzed objectively by using machine learning techniques.

North Korean Studies on Economic Policy
Outsiders can use a three-stage process to observe the performance of North Korean policy. The first stage is referred to as the planning phase, in which official propaganda and state media released by North Korean authorities, such as Rodong Sinmun, a daily newspaper in North Korea, and Economic Research, confirm that North Korean officials have reshaped or revised their policies. The second stage is the implementation of the policy, and is reflected in the actual application of the new policy. Due to the limitations of intelligence on North Korea, the actual execution is not clear. However, it is possible to verify the actual policies through the testimonies of North Korean defectors. The third stage is a performance realization step at which outsiders identify whether or not new policies result in different outcomes. It is quite challenging for outsiders to measure the actual policy effects, but partial confirmation is possible through evaluating the data compiled by international organizations, such as the Food and Agricultural Organization, the World Food Program, or the United Nations Development Program. Taking agricultural reform as an example, the first stage can be conjectured by reports and papers in Rodong Sinmun and Economic Research, respectively, on the agricultural reform. The second stage can be reflected in the agricultural reform being applied to cooperative farms in North Korea, and the third stage can be assessed using the data compiled by international organizations, whether or not actual agricultural production increased, due to the agricultural reforms. This study focuses on the first stage using economic papers officially published in Pyongyang.
Concerns about data shortage, accessibility, and credibility have always been raised in studies about the North Korean economy. Due to the nature of economics, the lack of statistics, in particular, inhibits the ability to analyze reforms and clearly serves as a limitation to any research. Despite these limitations, however, some scholars have attempted to estimate macroeconomic statistics on North Korea. Kim et al. [5], for example, calculated North Korea's gross national product and gross domestic product growth rates using official statistics from North Korea and South Korea. Some other studies explored changes in economic policies of the North Korean regime through state publications-although, these studies were conducted mainly by South Korean scholars who are able to access and read North Korean documents [6][7][8][9][10]. Choi and Lecy [6] divided their research into four periods (1986-1993, 1994-2001, 2002-2005, and 2006-2009), dependent on the policy direction, and presented a map of the inter-topic correlations shown in Economic Research. Park Jong-hui et al. [7] examined the subject changes of North Korea's New Year's speeches with automated text analysis techniques. Lee [8], Lim [9], and Na [10] also conducted in-depth text analyses of the primary literature, and publications and reports by the state media. In addition, Choi et al. [11] compared the industrial policies presented under the Kim Il-sung and Kim Jong-il administrations with the industrial policies emphasized by Kim Jong-un by conducting economic research on the Rodong Sinmun. Admittedly, these studies expanded upon the academic approaches and methodologies on North Korean studies, but they had limitations, such as the lack of objectivity and accuracy of the classification of topics, as well as the small number of documents that researchers could directly access and categorize.

Text Mining Approach
Though text analysis can be done manually, a more sophisticated analysis is possible using text mining techniques with computer algorithms. Text mining includes converting unstructured data and text into structured data for a computer to automatically analyze [12]. The translated text takes the same form as conventional big data, and by using machine learning techniques, researchers are able to analyze those reconstructed texts in various ways. There are limited cases in which text analysis utilizing in-depth text mining techniques are applied to North Korean literature. Park et al. [7], for instance, divided North Korea's New Year's messages by year to investigate annual changes in the direction of North Korea's policy by classifying similarities among the subjects that were extracted from keywords. Rich [13,14] analyzed the attitude of the six-party talks and U.S. foreign policy on the North Korean denuclearization issue in an English version of news reports released by the Korean Central News Agency. Whang et al. [15] compared patterns in the reports using a method of supervised learning, and examined military offenses to predict when North Korea would launch a small-scale military provocation.
There are many difficulties in analyzing North Korean texts. First of all, after more than 70 years of separation, the Korean language spoken and written in North Korea has become very different from that in South Korea. For example, there are different spellings, spacing standards, and double-tone rules. Second of all, certain words that are no longer used in South Korea are still used in North Korea, such as Gurapa for Europe or Rossia for Russia. Thus, the difference between the two Koreas makes morphological analysis-the most fundamental form of text analysis-difficult. Therefore, words that are not recognized by the morpheme analyzer tend to be treated as missing data. (A morpheme analyzer is built by an automated algorithm to learn a large volume of existing literature. A morphological analyzer for the North Korean language has not yet been established because of the lack of electronic data from North Korea).
To solve this problem, this study replaced the unique spelling and words of North Korea with similar words that can be analyzed. This is a method that has not been introduced in previous studies. In the absence of a morpheme analyzer for North Korean, replacing words that are not recognized by the morpheme analyzer with the closest alternative words can prevent any bias in the data, due to missing data. In fact, automated document analysis often results in a completely different outcome, even if only a few morphemes or words are missing. In particular, the accuracy of the analysis results can be greatly reduced if an unrecognized word is one that carries a significant amount of information instead of being one that is used habitually.

Economic Research, an Economics Journal in North Korea
The main texts for this study were extracted from North Korea's economics journal Economic Research. The period covered is between 1988 and 2018, which spans the ruling periods of Kim Il-sung, Kim Jung-il, and Kim Jong-un. The journal is a quarterly publication focusing on related economic issues, and was first published in 1956. The papers in Economic Research are written to discuss and disseminate North Korea's economic policies and future goals, as well as summarize international economic situations, not make academic contributions. Economic Research provides insight into the changes in North Korea's policies, institutions, and practices regarding economic policies, trade, and international financial issues. (Numerous academic journals are published in North Korea. They include Kyongje Yongu (Economic Research), Kimilseng Jonghaptayhak Hakpo (Academic Journal of Kim Il-sung University), Yeogsa Gwahag (Study of History), Chosun Uihak (Journal of Chosun Medicine), and Cheolhag Yeongu (Study of Philosophy). Among these journals, Economic Research is the only source of updated information about North Korea's economic policies and institutions).
While the New Year's address reveals the leader's intentions with regard to economic policies and goals, Economic Research represents a specific roadmap that is presented by experts across various sectors of the economy and written under the North Korean guidelines. Media outlets, such as the Rodong Sinmun and the Chosun Shinbo, share the intentions of the highest-ranking government officials to the public, but are not regarded as being as professional or as specific to economic issues as is Economic Research. In addition, the value and reputation of Economic Research as a professional journal in Pyongyang stems from its ability to disseminate various policy alternatives and economic policies of North Korea. To analyze the changes in economic policy topics in Economic Research, this paper divided the analysis period into three time periods: Kim Il-sung (1988-1994, Kim Jong-il (1995-2011, and Kim Jong-un (2012-2018). The reason for dividing the target period by the 'supreme leader' is that, firstly, there is no consensus in the academic community as to the period of division. Secondly, the distinction between periods under a specific policy agenda is expected to show whether a new regime enjoys any autonomy from its previous leader(s)' regimes, as North Korean people and government officials, who live under totalitarianism, have no choice but to follow the leader's instructions and guidance. Therefore, it is deemed appropriate to analyze the changes in policy subject matter by dividing the period by the leader.
Kim Jong-il appointed Kim Jong-un as his successor in 2009. Kim Jong-un was then appointed the Commanding General of the Korean People's Army in 2010, showing that a process of actual power inheritance started to take place. Table 1 shows that the number of papers published every year since 2009 increased by 10, and reached over 130 in 2012, reflecting a change in the composition of Economic Research.

Periods
Year

No. of Papers
Kim Il-sung   1988  27  1989  49  1990  64  1991  58  1992  64  1993  60  1994  47  subtotal  338   Kim Jong-il   1995  61  1996  59  1997  57  1998  58  1999  63  2000  66  2001  69  2002  71  2003  85  2004  79  2005  81  2006  85  2007 84 There is a kinship between the three leaders, who each inherited his predecessor's political and ideological ideas. For example, authentic socialist principles, such as 'self-reliance', based on the Juche ideology, and the 'strategic management by the state of the construction of a socialist economic powerhouse' are consistently emphasized. Nevertheless, clear distinctions between the leaders are observed from time to time. For instance, Kim Il-sung's main policy goals were to strengthen the rural economy under the socialist ideological framework, and establish an industrial base centered on small-and medium-sized businesses. Kim Jong-il introduced the capitalist system in North Korea during a rivalry with the capitalist South. Capitalism still exists under Kim Jong-un's leadership, but the views are less critical than they were under previous leadership. Remarkably, Kim Jong-un emphasized partial acceptance of the capitalist system for the sake of economic revival, including science, information, a knowledge-based economy and trade, and the so-called industrial revolution in the new century. Furthermore, academic journals reflect different tones depending on the leader at the time of publication. For example, during the leadership of Kim Jong-il in 1995, a paper containing critical details of the capitalist system was published at the end of every quarter. In addition, the column entitled Sangsik ('Common Sense' in English), which has been included at the end of each issue since 2010, is a structure designed to introduce basic economic concepts, such as knowledge, labor, the balance of payments, interest rates and depreciation. Consequently, some papers contain introductions, as well as criticisms about the market economy and liberal economics.

Main Results from Economic Research Using Text Mining Approach
In this section, the titles of the articles in Economic Research are analyzed to verify whether the characteristics shown in the titles of the paper can be classified by leader or time. After the text is converted into structured data, machine learning is used to analyze and validate text as the dimensions in the dataset increase, and it, thus, becomes big data. This process represents the probabilistic or non-probabilistic word representation of a document or word as a vector. In this section, we mobilize a term frequency-inverse document frequency (tf-idf) process, a method of non-probabilistic representation, and a machine learning classification technique to break down the main themes of economic policy in the title by time, subject, and keyword [12].
First, we look at how year-by-year article titles are clustered by subject by using unsupervised learning without prior information on the characteristics of the title. If a cluster of titles is consistent within the same time span, it can be implied that the focus and policy direction of the economy depends on the leader at that time. Second, it predicts when individual papers were written by using supervised learning, and reviews them to see if individual titles can be classified according to the characteristics.
Knowing when a paper was published could provide further evidence of the distinct characteristics of the economic policy direction in the concerned time period. Third, we provide a visualized frequency of the keywords of the thesis titles, depending on each leader's period. We also look at the major topic changes between each period using a latent Dirichlet allocation (LDA), which extracts the subject from the Bayesian probability process. Figure 1 displays our research process in a flowchart.
Sustainability 2020, 12, x FOR PEER REVIEW 7 of 24 supervised learning, and reviews them to see if individual titles can be classified according to the characteristics. Knowing when a paper was published could provide further evidence of the distinct characteristics of the economic policy direction in the concerned time period. Third, we provide a visualized frequency of the keywords of the thesis titles, depending on each leader's period. We also look at the major topic changes between each period using a latent Dirichlet allocation (LDA), which extracts the subject from the Bayesian probability process. Figure 1 displays our research process in a flowchart.

Keyword Correlation Coefficient for Each Year
The information characteristics of the data are tested before using tools, such as machine learning. Firstly, the title words in Economic Research for each year are represented in a Term-Document Matrix (TDM), and secondly, the correlation coefficient of the title of the document by year is calculated (see Figure 2). If the value of the coefficient in the title for each year is 0.6 or higher, the topics covered in Economic Research can be considered to have a high level of similarity. However, there are limitations to only using the value of the correlation coefficient to analyze how the topics of interest in economic policies change over time. For example, the large number of keywords commonly used in the title of the papers in the journal make using only this method inefficient. Therefore, it would be a more efficient method of analysis to determine and classify the weights of common words using a tf-idf process.

Keyword Correlation Coefficient for Each Year
The information characteristics of the data are tested before using tools, such as machine learning. Firstly, the title words in Economic Research for each year are represented in a Term-Document Matrix (TDM), and secondly, the correlation coefficient of the title of the document by year is calculated (see Figure 2). If the value of the coefficient in the title for each year is 0.6 or higher, the topics covered in Economic Research can be considered to have a high level of similarity. However, there are limitations to only using the value of the correlation coefficient to analyze how the topics of interest in economic policies change over time. For example, the large number of keywords commonly used in the title of the papers in the journal make using only this method inefficient. Therefore, it would be a more efficient method of analysis to determine and classify the weights of common words using a tf-idf process.

Categorization of the Title of Each Year's Thesis
Applying unsupervised learning to draw inferences from titles that are not randomly classified in advance is the first step in determining whether information can be extracted from the characteristics of the year-by-year thesis. There are many different methods of unsupervised learning taxonomy, but the algorithm used in this study is simple and intuitive; the hierarchical clustering method, which can group data that have not been classified according to interconnections, is used. In summary, it can be inferred that, when the title of the annual paper is clustered according to shared attributes, and the cluster coincides with the leadership period of a leader, the characteristics are shown in the title represent a change in the economic policy made by the leader.

Categorization of the Title of Each Year's Thesis
Applying unsupervised learning to draw inferences from titles that are not randomly classified in advance is the first step in determining whether information can be extracted from the characteristics of the year-by-year thesis. There are many different methods of unsupervised learning taxonomy, but the algorithm used in this study is simple and intuitive; the hierarchical clustering method, which can group data that have not been classified according to interconnections, is used. In summary, it can be inferred that, when the title of the annual paper is clustered according to shared attributes, and the cluster coincides with the leadership period of a leader, the characteristics are shown in the title represent a change in the economic policy made by the leader.  The first subcluster of the second cluster in the middle of the dendrogram (1988-1998) covers the period of Kim Il-sung's leadership (1988)(1989)(1990)(1991)(1992)(1993)(1994). In this subcluster, the similarity between annual titles is very high. First of all, compared with North Korea's economic growth rate, as shown in Figure  4, it can be seen that the first cluster is within the period that the North Korean regime was in crisis, during which the economy was designated as a default country in 1987 and experienced negative growth. Second of all, it can be inferred that the topic of interest in the economic policies during the  The first subcluster of the second cluster in the middle of the dendrogram (1988-1998) covers the period of Kim Il-sung's leadership (1988)(1989)(1990)(1991)(1992)(1993)(1994). In this subcluster, the similarity between annual titles is very high. First of all, compared with North Korea's economic growth rate, as shown in Figure 4, it can be seen that the first cluster is within the period that the North Korean regime was in crisis, during which the economy was designated as a default country in 1987 and experienced negative growth. Second of all, it can be inferred that the topic of interest in the economic policies during the Kim Il-sung era was maintained with no major changes. That is, early in his leadership (1995-1997), Kim Jong-il followed his father's policy direction without making any major changes. The first subcluster of the second cluster in the middle of the dendrogram (1988-1998) covers the period of Kim Il-sung's leadership (1988)(1989)(1990)(1991)(1992)(1993)(1994). In this subcluster, the similarity between annual titles is very high. First of all, compared with North Korea's economic growth rate, as shown in Figure  4, it can be seen that the first cluster is within the period that the North Korean regime was in crisis, during which the economy was designated as a default country in 1987 and experienced negative growth. Second of all, it can be inferred that the topic of interest in the economic policies during the Kim Il-sung era was maintained with no major changes. That is, early in his leadership (1995-1997), Kim Jong-il followed his father's policy direction without making any major changes. The second subcluster reflects a change in policy direction, which was triggered by the North Korean government, in a state of crisis, allowing some marketization in its economic system from 1999. It is also speculated that this period was followed by South Korea's The second subcluster reflects a change in policy direction, which was triggered by the North Korean government, in a state of crisis, allowing some marketization in its economic system from 1999. It is also speculated that this period was followed by South Korea's policy of reconciliation and cooperation with the North, known as the Sunshine Policy, which contributed to the recovery of the North Korean economy, despite much criticism that the policy led to the North developing nuclear weapons and subsequently worsening the national security issue [16,17]. Growth started to rise sharply, but the North Korean authorities began to curb the expansion of marketization from the bottom in 2006. These socioeconomic changes in North Korean society are reflected in the second stage of the second subcluster of the dendrogram (2006-2010) in Figure 3.
The left-hand cluster of the dendrogram, the first cluster, is the Kim Jong-un period (2011-2018), which is divided into two subclusters, 2011-2016 and 2017-2018. The results of the clustering show that the Kim Jong-un period is not similar to the previous periods. (The vertical height of the stem in the dendrogram indicates the cosine similarity between titles by year. The higher the height, the lower the similarity. The Kim Il-sung and Kim Jong-il periods are also clustering, but in terms of similarity, they are closer to each other than they are to the Kim Jong-un period). This suggests that there have been notable changes in policy direction and issues of interest compared to the earlier eras. In fact, Kim Jong-un, who has been in power since 2011, began to take full advantage of the market when exports and imports greatly increased. As a result, the growth rate shifted upward, but the North Korean economy once again experienced negative growth in the second subcluster (2017-2018) after the United Nation Security Council imposed tougher sanctions on North Korea. Figure 3 shows that the topics in Economic Research are clearly distinct from those in any other period.

Predicting a Paper's Publishing Year Using Logistic Regression
By utilizing a supervised learning approach, one of the machine learning classification techniques, this section verifies whether the timing of the appearance of keywords can be predicted by the characteristics found in the title of papers. Supervised learning techniques include logistic regression, support vector machines, and naïve Bayesian classifiers. This study predicts when each paper published in Economic Research was written based on the logistic regression, which shows the highest probability of classification among the various classification techniques (see Appendix A.1).
First, the documents that have undergone pre-processing and non-probability word representation are a set of multidimensional vectors. The total number of main words used in this study is 1846, so each title (2757 in total) is represented as a vector of 1 × 1846 dimension (nouns that appear more than once in every article title are called 'vocabulary' or 'dictionary'). The dimension of the DTM or tf-idf matrix is 2757 × 1846. Since the sample size is relatively small (this has a relatively small dimension compared to the sample size of so-called big data, which has been actively researched recently), the characteristics of the titles of the papers are distinguished by the leader's period of leadership when the accuracy of the classification through the machine learning algorithm is high.
Based on the results of hierarchical clustering, the categories were divided into four scenarios: 1. Kim Il-sung, Kim Jong-il, Kim Jong-un; 2. Kim Il-sung, Kim Jong-il; 3. Kim Il-sung, Kim Jong-un; and 4. Kim Jong-il and Kim Jong-un. Since the number of subject words is 1846, there are too many characteristics or variables for each title. We, therefore, reduce the dimension of the characteristics through a principal component analysis (PCA). We set the dimension to obtain an optimal number of characteristics up to 300. The logistic regression with L2 regularization selects the most relevant features of the 300. In addition, the k-overlapping cross-checking described earlier evaluated the performance of the parent model, selected the optimal model, and checked the classification performance using selected hyperparameters and rating indicators at a level that minimizes the overfitting and underfitting dilemmas. The hyperparameter used for cross-checking is: c = 10 −4 , 10 −3 , 10 −2 , 10 −1 , 10 1 , 10 2 , 10 3 . (1) Classifications of the eras of Kim Il-sung, Kim Jong-il, and Kim Jong-un. Figure 5 shows the results of the cross-examination from classifying the titles published during the leadership of Kim Il-sung, Kim Jong-il, and Kim Jong-un. To classify the titles by logistic regression, 1846 dimensions of the rare tf-idf matrix were reduced to 300 dimensions by PCA. (The number of features to be classified corresponds to a hyperparameter. In this case, the classification was classified into various numbers ranging from 3 to 300, but the accuracy of the classification was inferior to the characteristics of a number that is too low. Therefore, 300 characteristics or dimensions are implied to reflect the most abundant characteristics). The cross-validation test in Figure 5 confirms that there is little overfitting, despite it mitigating the constraints of the model. While the risk of overfitting is not significant, the appropriate modeling constraint is c = 100, at a level that minimizes underfitting. Here, the accuracy of the training data prediction is 0.77, and the subject of the paper published in Economic Research throughout the three leaders' periods can be seen as having relatively distinct characteristics. The accuracy of the prediction (if there is more than one category, evaluation indicators other than precision are omitted because their meaning is unknown) is 0.64, and each published title in Economic Research alone can predict, to some extent, when the paper was written. Therefore, over the three time periods, the main concerns and policy directions of the economy are reflected in the titles of the papers. throughout the three leaders' periods can be seen as having relatively distinct characteristics. The accuracy of the prediction (if there is more than one category, evaluation indicators other than precision are omitted because their meaning is unknown) is 0.64, and each published title in Economic Research alone can predict, to some extent, when the paper was written. Therefore, over the three time periods, the main concerns and policy directions of the economy are reflected in the titles of the papers.  Figure 6 shows the accuracy ratio of the results predicted by the title of the paper and the confusion matrix. The diagonal elements in the confusion matrix are the probability that the predicted results are true, and the closer we get to one, the more distinct the characteristics are. When the Kim Il-sung, Kim Jong-il, and Kim Jong-un periods are grouped together, the results of the test data predict (this means the out of sample forecast discussed in econometrics) that a thesis title in the Kim Il-sung time period (class 0) is written in the Kim Il-sung period with 80% accuracy. The results also predict that a thesis title in the Kim Jong-il time period (class 1) is written in the Kim Jong-il period with 53% accuracy, and a thesis title in the Kim Jong-un time period (class 2) is written in the Kim Jong-un period with 61% accuracy. These accuracies were lower than those in the Kim Il-sung period, implying that the farther from the diagonal element of the confusion matrix, the lower the level of error. Therefore, it can be said that the characteristics of a paper title are clearly distinguished during the leadership periods of Kim Il-sung and Kim Jong-un. However, there is a 15-29% discrepancy between the titles of the papers between the Kim Il-sung and Kim Jong-il periods, and between the Kim Jong-il and Kim Jong-un periods.  Figure 6 shows the accuracy ratio of the results predicted by the title of the paper and the confusion matrix. The diagonal elements in the confusion matrix are the probability that the predicted results are true, and the closer we get to one, the more distinct the characteristics are. When the Kim Il-sung, Kim Jong-il, and Kim Jong-un periods are grouped together, the results of the test data predict (this means the out of sample forecast discussed in econometrics) that a thesis title in the Kim Il-sung time period (class 0) is written in the Kim Il-sung period with 80% accuracy. The results also predict that a thesis title in the Kim Jong-il time period (class 1) is written in the Kim Jong-il period with 53% accuracy, and a thesis title in the Kim Jong-un time period (class 2) is written in the Kim Jong-un period with 61% accuracy. These accuracies were lower than those in the Kim Il-sung period, implying that the farther from the diagonal element of the confusion matrix, the lower the level of error. Therefore, it can be said that the characteristics of a paper title are clearly distinguished during the leadership periods of Kim Il-sung and Kim Jong-un. However, there is a 15-29% discrepancy between the titles of the papers between the Kim Il-sung and Kim Jong-il periods, and between the Kim Jong-il and Kim Jong-un periods.  Kim Il-sung and Kim Jong-il time period comparisons. The accuracy of predictions will be greatly improved if the characteristics of the titles classified as only Kim Il-sung and Kim Jong-il are compared. The cross-validation test in Figure 7 shows no overfitting, while the modal constraint of underfit improves to a low level, at c = 10 . (This shows that many of the characteristics of the 300 dimensions contribute to the classification). The accuracy of the classification of training data at the constraint level of c = 10 is 0.92, and the subjects of the papers published in Economic Research during Kim Il-sung and Kim Jong-il's leadership periods have distinct characteristics. As shown in year-by-year clusters, Kim Jong-il initially inherited the policies of his father Kim Il-sung, but pursued his own, due to changes in the economic environment and potential national security threats at home and abroad. The confusion matrix between Kim Il-sung and Kim Jong-un is the same as in Figure 8. The accuracy of predictions for Kim Il-sung (class 0) and Kim Jong-il (class 1) periods in the paper titles is 82% for both. The recall, precision, and F1 scores are also seen to be very decent at predicting in both periods, and are 0.82, 0.81, and 0.82, respectively. The confusion matrix between Kim Il-sung and Kim Jong-un is the same as in Figure 8. The accuracy of predictions for Kim Il-sung (class 0) and Kim Jong-il (class 1) periods in the paper titles is 82% for both. The recall, precision, and F1 scores are also seen to be very decent at predicting in both periods, and are 0.82, 0.81, and 0.82, respectively. The results show that there has been a major shift in economic policy and the economic environment between the two generations. The results of cross-checking are shown in Figure 9. Even the regulatory parameter at the c = 10 level is significantly improved, and the predicted accuracy of training data is close to one. Accordingly, the papers published in Economic Research during the Kim Il-sung and Kim Jong-un periods can be predicted with very high accuracy. The results show that there has been a major shift in economic policy and the economic environment between the two generations. The results of cross-checking are shown in Figure 9. Even the regulatory parameter at the c = 10 5 level is significantly improved, and the predicted accuracy of training data is close to one. Accordingly, the papers published in Economic Research during the Kim Il-sung and Kim Jong-un periods can be predicted with very high accuracy.
Kim Il-sung and Kim Jong-un time comparisons. The accuracy of predictions is improved even further if the characteristics of the titles classified as only Kim Il-sung and Kim Jong-un are compared. The results show that there has been a major shift in economic policy and the economic environment between the two generations. The results of cross-checking are shown in Figure 9. Even the regulatory parameter at the c = 10 level is significantly improved, and the predicted accuracy of training data is close to one. Accordingly, the papers published in Economic Research during the Kim Il-sung and Kim Jong-un periods can be predicted with very high accuracy. The confusion matrix between Kim Il-sung and Kim Jong-un are shown in Figure 10. The prediction accuracy of the Kim Il-sung period (class 0) and the Kim Jong-un period (class 1) for the test data are 97% and 85%, respectively, and the overall accuracy is 0.91. The recall, precision, and F1 scores are also seen to be very decent at predicting the model, and are 0.85, 0.96, and 0.90, respectively. The confusion matrix between Kim Il-sung and Kim Jong-un are shown in Figure 10. The prediction accuracy of the Kim Il-sung period (class 0) and the Kim Jong-un period (class 1) for the test data are 97% and 85%, respectively, and the overall accuracy is 0.91. The recall, precision, and F1 scores are also seen to be very decent at predicting the model, and are 0.85, 0.96, and 0.90, respectively.  Figure 11. The level of the regulation parameter at c = 10 is improving, but there is no indication of overfitting. The precision of the training data prediction at c = 10 is 0.82. The confusion matrix between Kim Jong-il and Kim Jong-un is shown in Figure 12. The prediction accuracy of the paper titles being in the respective Kim Jong-il (class 0) and Kim Jong-un  Figure 11. The level of the regulation parameter at c = 10 is improving, but there is no indication of overfitting. The precision of the training data prediction at c = 10 is 0.82. The confusion matrix between Kim Jong-il and Kim Jong-un is shown in Figure 12. The prediction accuracy of the paper titles being in the respective Kim Jong-il (class 0) and Kim Jong-un (class 1) periods are 68% and 71%, respectively. The recall, precision, and F1 scores are 0.71, 0.69, and 0.70, respectively, and the overall prediction accuracy for the model is 0.70. Therefore, the prediction is relatively decent.  Figure 12. The prediction accuracy of the paper titles being in the respective Kim Jong-il (class 0) and Kim Jong-un (class 1) periods are 68% and 71%, respectively. The recall, precision, and F1 scores are 0.71, 0.69, and 0.70, respectively, and the overall prediction accuracy for the model is 0.70. Therefore, the prediction is relatively decent.

Changes in Economic Environment and Policy Topics by Leader
Some keywords regarding socialist systems, criticisms of capitalism, and encouragement for the people frequently appeared throughout the Kim Il-sung, Kim Jong-il, and Kim Jong-un periods. As Figure 2 shows, there is a high correlation coefficient of paper topics. Nevertheless, words related to topics of particular interest in a period are expected to appear more frequently in the titles in that period than in other periods. Taking these aspects into account, we look at the big picture of North Korea's economic policy changes by analyzing the differences between each period based on the appearance and frequency of certain keywords.
Frequency of keywords by time period. Figure 13 shows the frequency of keywords during Kim Il-sung's leadership period (1988)(1989)(1990)(1991)(1992)(1993)(1994). The most commonly appeared words or clichés, such as 'critical tasks', 'essential requirements', 'reactions', 'problems', 'preconditions', 'important principles', and 'struggle tasks', are excluded for being stopwords (When calculating the sample frequency, the common words may be treated as stopwords, but if the common words are also used as meaningful words in the title, they cannot be treated as stopwords. In the case of the tf-idf process described above, as the weight for any particular common word approaches zero, that common word is naturally removed). However, common words that were significant in a given context were not deleted.
If the frequency of keywords, such as 'technology', 'labor', 'rural areas', 'agriculture', 'industry and capitalism', is high, we assume that, at the time, the regime was shifting the direction of economic policy toward industrialization. Even though it criticized the capitalist system, it was concerned about labor productivity, technology, and capitalism, and was eager to strengthen its agricultural production.

Changes in Economic Environment and Policy Topics by Leader
Some keywords regarding socialist systems, criticisms of capitalism, and encouragement for the people frequently appeared throughout the Kim Il-sung, Kim Jong-il, and Kim Jong-un periods. As Figure 2 shows, there is a high correlation coefficient of paper topics. Nevertheless, words related to topics of particular interest in a period are expected to appear more frequently in the titles in that period than in other periods. Taking these aspects into account, we look at the big picture of North Korea's economic policy changes by analyzing the differences between each period based on the appearance and frequency of certain keywords.
Frequency of keywords by time period. Figure 13 shows the frequency of keywords during Kim Il-sung's leadership period (1988)(1989)(1990)(1991)(1992)(1993)(1994). The most commonly appeared words or clichés, such as 'critical tasks', 'essential requirements', 'reactions', 'problems', 'preconditions', 'important principles', and 'struggle tasks', are excluded for being stopwords (When calculating the sample frequency, the common words may be treated as stopwords, but if the common words are also used as meaningful words in the title, they cannot be treated as stopwords. In the case of the tf-idf process described above, as the weight for any particular common word approaches zero, that common word is naturally removed). However, common words that were significant in a given context were not deleted.
If the frequency of keywords, such as 'technology', 'labor', 'rural areas', 'agriculture', 'industry and capitalism', is high, we assume that, at the time, the regime was shifting the direction of economic policy toward industrialization. Even though it criticized the capitalist system, it was concerned about labor productivity, technology, and capitalism, and was eager to strengthen its agricultural production. Figure 14 shows the frequency of certain keywords during Kim Jong-il's leadership period (1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011). First, we can see that the frequency of 'technology' is as high as during his predecessor's leadership period, and some of the frequently-appearing words are 'capitalism', 'science', 'information', 'management', 'strong state', 'modernization', 'industry', 'international relations' and related words. During the Kim Jong-il period, there are many papers criticizing the capitalist system, while propagandizing the socialist system amid the rivalry with the South. In particular, at the end of each issue, one or two papers are classified as making criticisms of the capitalist system. At the same time, however, modern business management and economic management methodologies, such as international capital markets, market development, and increases in trade volume, are actively introduced amid economic hardships, when revitalizing the socialist economy is a goal.

deleted.
If the frequency of keywords, such as 'technology', 'labor', 'rural area', 'agriculture', 'industry and capitalism', is high, we assume that, at the time, the regime was shifting the direction of economic policy toward industrialization. Even though it criticized the capitalist system, it was concerned about labor productivity, technology, and capitalism, and was eager to strengthen its agricultural production.  During the Kim Jong-un leadership period (2012-2018), there is a high frequency of words, such as 'production', 'knowledge', 'technology', 'power', 'information', and 'science'. Under the Kim Jong-un regime, the weight of titles related to criticisms of the capitalist system decreased significantly. However, the emergence of words like 'improving productivity', 'knowledge', and 'technology' indicates that economic policies are turning to economic growth based on pragmatism.
Sustainability 2020, 12, x FOR PEER REVIEW 16 of 25 Figure 14 shows the frequency of certain keywords during Kim Jong-il's leadership period (1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011). First, we can see that the frequency of 'technology' is as high as during his predecessor's leadership period, and some of the frequently-appearing words are 'capitalism', 'science', 'information', 'management', 'strong state', 'modernization', 'industry', 'international relations' and related words. During the Kim Jong-il period, there are many papers criticizing the capitalist system, while propagandizing the socialist system amid the rivalry with the South. In particular, at the end of each issue, one or two papers are classified as making criticisms of the capitalist system. At the same time, however, modern business management and economic management methodologies, such as international capital markets, market development, and increases in trade volume, are actively introduced amid economic hardships, when revitalizing the socialist economy is a goal.
During the Kim Jong-un leadership period (2012-2018), there is a high frequency of words, such as 'production', 'knowledge', 'technology', 'power', 'information', and 'science'. Under the Kim Jongun regime, the weight of titles related to criticisms of the capitalist system decreased significantly. However, the emergence of words like 'improving productivity', 'knowledge', and 'technology' indicates that economic policies are turning to economic growth based on pragmatism. Topic models for economic policy by time period. In this section, the LDA method is used to conjecture keywords and the topic of interest for each leader's leadership period. An LDA [18] method is a probability model that classifies the topics of a document based on the probability that a particular document is a mixture of a small number of topics, and that each word is attributable to one of the document's topic [12]. The probabilistic model of the LDA method shows the most appropriate topic and probability according to the document or document set by estimating the maximum probability of there being a combination of the words and topics. In this paper, the hyperparameter representing the maximum number of topics is limited to 25. As a result of the estimation, similar topics are represented as one topic, and so the number of topics for each period can be determined as such. Table 2 is the result of selecting a topic based on the keywords of each topic as a result of the LDA estimation. As can be seen from the table, the distribution of topics varies as the Kim Jong-un period progresses. Meanwhile, there are many topics on criticizing the capitalist system and promoting the construction of a communist society in the Kim Il-sung period.  Topic models for economic policy by time period. In this section, the LDA method is used to conjecture keywords and the topic of interest for each leader's leadership period. An LDA [18] method is a probability model that classifies the topics of a document based on the probability that a particular document is a mixture of a small number of topics, and that each word is attributable to one of the document's topic [12]. The probabilistic model of the LDA method shows the most appropriate topic and probability according to the document or document set by estimating the maximum probability of there being a combination of the words and topics. In this paper, the hyperparameter representing the maximum number of topics is limited to 25. As a result of the estimation, similar topics are represented as one topic, and so the number of topics for each period can be determined as such. Table 2 is the result of selecting a topic based on the keywords of each topic as a result of the LDA estimation. As can be seen from the table, the distribution of topics varies as the Kim Jong-un period progresses. Meanwhile, there are many topics on criticizing the capitalist system and promoting the construction of a communist society in the Kim Il-sung period. There was a growing interest in the international economy during the Kim Jong-il period, but, ironically, coincided with criticisms of 'Western imperialism'. Kim Jong-un's period deals with a wide range of topics that differ from his predecessors, depicted in the research with words, such as 'power of knowledge', and 'power of science', 'modern management', 'economics', and 'statistics', as well as 'the international financial market'. More weight is being placed on practical economic reform ahead of the 'ideology of successful construction of a communist or socialist economy' than before.

Overview of Kim Jong-un's Economic Strategy
Considering the above analysis, the policy directions, subjects, and environment in the Kim Jong-un period are clearly different from those of previous periods. Thus, it is necessary to examine the papers in Economic Research in this period in more detail. First, in terms of composition and form, the number of publications has increased remarkably since Kim Jong-un became leader, and the topics and researchers covered by the paper have diversified. Second, it can be inferred from this change that policymakers and economists in the country are learning about the open capitalist economy.
As Figure 15 shows, the frequency of different topics between the Kim Il-sung and Kim Jong-il periods and the Kim Jong-un period, such as production, knowledge, and technology, is increasing. Despite inheriting the 'self-reliance' ideology, Kim Jong-un is pursuing a different path than his predecessors did to rebuild the North Korean economy, as announced in the 2016 New Year's address, which focused on science and technology advancements and the achievement of a 'nuclear state, as shown in Table 2. The policies adopted by Kim Jong-un promote a less closed or alienated economic system than those his predecessors favored. In other words, major keywords, such as 'foreign banking systems', 'currency and exchange rates', 'trade theory', 'competitiveness in the era of internationalization', and 'overseas banking system' show that Kim Jong-un aims to expand and develop engines of economic growth similar to those used in the Western world, and demands that knowledge and technology realize his plans. economy.
As Figure 15 shows, the frequency of different topics between the Kim Il-sung and Kim Jong-il periods and the Kim Jong-un period, such as production, knowledge, and technology, is increasing. Despite inheriting the 'self-reliance' ideology, Kim Jong-un is pursuing a different path than his predecessors did to rebuild the North Korean economy, as announced in the 2016 New Year's address, which focused on science and technology advancements and the achievement of a 'nuclear state, as shown in Table 2 . The policies adopted by Kim Jong-un promote a less closed or alienated economic system than those his predecessors favored. In other words, major keywords, such as 'foreign banking systems', 'currency and exchange rates', 'trade theory', 'competitiveness in the era of internationalization', and 'overseas banking system' show that Kim Jong-un aims to expand and develop engines of economic growth similar to those used in the Western world, and demands that knowledge and technology realize his plans. Additionally, from 2017, 'Silmari', which corresponds to 'Keywords' in English, was added at the end of each paper. For example, if the title of the paper is, 'Our economic management method that embodies the construction of economic power and the self-reliance ideology', then 'economic power', 'self-reliance', and 'economic management' are selected as 'Silmari'. This new addition means that managing keywords more systematically can make searching for related topics more efficient. The topics of the latest paper include 'knowledge economy', 'informationization', 'intellectual property rights', 'science and technology', 'tourism', 'economic and trade zones', 'special economic development zones', 'foreign exchange rates', 'accounting', 'finance', 'international banks', and 'Public-Private Partnerships (PPPs)', among others. There are two main reasons for this. First, North Korea's economy has not improved much since Kim Jong-un took power in 2012. The economic growth rate was 1.1% in 2013, and the economy retracted by 3.5% and 4.1% in 2017 and 2018, respectively, shortly after the ambitious New Strategy was launched. However, illegal and unofficial economic activities through the market have exposed Additionally, from 2017, 'Silmari', which corresponds to 'Keywords' in English, was added at the end of each paper. For example, if the title of the paper is, 'Our economic management method that embodies the construction of economic power and the self-reliance ideology', then 'economic power', 'self-reliance', and 'economic management' are selected as 'Silmari'. This new addition means that managing keywords more systematically can make searching for related topics more efficient. The topics of the latest paper include 'knowledge economy', 'informationization', 'intellectual property rights', 'science and technology', 'tourism', 'economic and trade zones', 'special economic development zones', 'foreign exchange rates', 'accounting', 'finance', 'international banks', and 'Public-Private Partnerships (PPPs)', among others. There are two main reasons for this. First, North Korea's economy has not improved much since Kim Jong-un took power in 2012. The economic growth rate was 1.1% in 2013, and the economy retracted by 3.5% and 4.1% in 2017 and 2018, respectively, shortly after the ambitious New Strategy was launched. However, illegal and unofficial economic activities through the market have exposed weaknesses in the state management of economic resources. In addition, corruption, which has spread through illegal economic activities, is posing a threat to the centrally-planned economy by distorting the allocation of resources and threatening the socialist principle itself.

Economic Strategies and the Roles of the State
Second, the most imminent task for the regime is the restoration of authority and control of the state through 'economic management'. Since Kim Il-sung's death in 1994, North Korea has been suffering from an economic crisis, as well as a gradual weakening in the trust and authority of the supreme leaders, Kim Il-sung and Kim Jung-il. Under these circumstances, the prolonged economic hardships have led to a vicious cycle of distrust in the state's functions and authority. Therefore, the first task of the Kim Jong-un regime was to restore the authority of the state (or of the Kim family), along with economic rehabilitation. Under this backdrop, the state's control of economic management, which the North Korean authorities emphasize, is quite clearly delineated.
An article entitled, 'The Status and Role of the State for Economic Management' published in the first issue of 2018, for instance, defines the status and the roles of the state as follows: "The nation's strategic management of economic projects means that the state functions in a commanding role to develop the economy, while actively taking various economic and organizational measures with a clear economic development vision". It also specifically describes three core roles of the state: First, 'achieving socioeconomic development based on quantitative growth of production'; second, 'defining the direction of the national economy with specific goals and development strategies'; and third, 'commanding and controlling the overall economic systems based on these development strategies'.
However, the following comments on the change in the role of the 'state', which distinguishes it from the previous Kim Il-sung and Kim Jong-il periods, are noteworthy: "The strategic management of the country's economic projects requires the state to map core sectors and core units of production and management activities, not all sector or units, but, to focus its [state] capabilities on its core areas to activate the entire economy" The strategy of the 'state' not controlling all sectors and all units, but focusing its core capabilities on key areas, can be interpreted as a deviation from the centralized state control system of an authentic socialist economy.
This subtle change accepted partial failures of the socialist economy, leading to the emergence of a new management style at state-run enterprises, the 'independent management system', in the early 2000s. Due to the breakdown of the national public distribution system, and the collapse of production facilities and capacity, North Korean authorities at that time ordered state-owned enterprises to be managed by individual companies to encourage 'the autonomy of the enterprise' and support senior managers and operations. This, in turn, meant the state could avoid the responsibility of running state-owned enterprises. There is a dilemma for the Kim Jong-un regime, which seeks to maintain the control function of the state, while responding to the damage that state authority caused through marketization from below. Internally, the Kim Jong-un regime could face several constraints, such as the loss of momentum for economic development, the weakening of state authority, and the deepening of economic dependence on China, due to economic sanctions.
Acknowledging this current policy environment and the economic situation in crisis, Kim Jong-un proposed holding the party congress and said, "(The) party will present a new five-year plan for national economic development, including the direction of the project in the next year, as well as the project of the party's central committee this year." The following sections elaborate specifically on strategies for the industrial sector and economic measures, to overcome these internal and external challenges.

Characteristics by Industrial Sector
By looking at the industrial strategies the Kim Jong-un regime initiated by sector, we can see the specific direction and goals of the economic strategy for which North Korea is aiming. Industry-related keywords mainly featured in the Kim Jong-un regime are 'the age of the knowledge economy' and 'construction of a strong science and technological state', along with those that represent maintaining the strategic status of the heavy and defense industries. By stressing that 'science and technology is the nation's most important strategic resource', it has set 'all of science and technology and production' as its policy goals.
To adapt to the age of the knowledge economy, in which knowledge is 'utilized as a basic source of production in the resource-intensive machine manufacturing industry', the regime stresses that the integration of science and technology and production and standardization should be made at production facilities to establish 'standardization in the production system, so the integration of human resources, research and development, production, sales, and even exports, can be used to build a production base for high-tech products. It also proposes specific measures, such as achieving 'modernization of the people's economy' and 'informationization' through the development of science and technology, and in particular, the selection of talented workers for training professional, scientific personnel and educating at central institutions. Under the slogan of the 'establishment of a research-oriented university' and 'creative talent development', Kim Jong-un instructed that major universities, including Kim Il-sung University, be transformed into research-oriented universities to meet the needs of the 'knowledge economy era'.
The papers published in 2016 and 2017 need more attention. Before and after the announcement of the New Strategy in 2016, Kim Jong-un seemed to feel that North Korea had escaped the worst economic difficulties, and declared that fostering the tourism industry, attracting overseas investment, and developing special economic zones would be the main drivers of national development and restoring the country's authority. Specifically, the regime intended to boost the tourism industry by modernizing tourism facilities, industrializing hotel services, designating tourist development zones, and constructing ecological tourism bases. This strategy is seen to have been an attempt to resolve the foreign currency shortage, as the tourism industry is recognized as an exception from the economic sanctions.
The importance of attracting foreign capital and developing special economic zones during this period is being reflected in every issue of Economic Research. This is an external economic strategy that is very comparable to the Kim Il-sung and Kim Jong-il periods, and almost every volume now contains an introduction to international financial markets and foreign trade. It is worth noting that international financial market papers, for example, share very specific knowledge, such as hedging methods, accounting verification of foreign investment entities, credit at international banks, investment inducement plans, government and joint investment methods for foreign companies, tax credits, and tax favors for foreign companies.
Another notable feature is the emergence of papers related to credit transactions and exchange rate forecasts, which can be said to be a key to economic growth forecasts and the operation of international capital markets, as well as the introduction of foreign exchange account preparation methods and econometric models. For example, since the mid-2000s, there has been an increase in the number of papers that introduced basic aspects of finance and accounting, such as 'the general principles of statistical data collection' and 'International Accounting Standards', measuring derivatives trading and hedging, factors affecting the exchange rate, forecasting management statistics, methods for strategic preparation of national foreign exchange accounts, and methods for determining overdue collateral tax, thereby requiring more specific knowledge and techniques of the reader. Despite the level of quality, we notice that these introductions to capitalist market economic systems aim to educate policymakers and economists.

Prospect and Managerial Issues of Economic Reform in the North Korean Economy
As analyzed above, the North Korean economy of Kim Jong-un regime is aimed at expanding its openness to the outside world, realizing science and knowledge economy, and improving the standard of people's lives, while sticking to the Juche ideology and the planned economy centered on the state and the party. The question is whether the regime has the resources and national capacity to actually implement its economic strategies. Among the various studies related to national competence, recent studies have highlighted authority, acceptance, and ability as important factors. These three A's were initially used as analytical frameworks in various areas, including the study of the acceptance of information and technology [19], the communication between medical clinicians and patients [20], and the authority and acceptance of performance evaluation by employees in corporate management [21], and have recently been used as major variables in the execution of public policies [22,23].
An analysis of North Korea's economic policy implementation capabilities based on these three factors shows that it is not so optimistic. While the authority by the state and the party is alive in a situation where the public control and surveillance system by the state, and the party still operates strongly, reform by 'authoritarianism without authority' is not sustainable. As seen in a study of Russia by Stoner-Weiss [24], 'authoritarianism without authority' can only worsen political corruption and social gaps. This weakening of authority will make the general public less receptive to policy, while increasing distrust in the state, and in the end, the authority will never be able to generate the policy effect intended by the state. Even under the Kim Jong-un regime, the nation's public economic system is not working properly, which further expands the informal economy and the illegal market economy. The growing informal economy, resulting from this weakening of the public sector, is a trigger to undermine the authority of the state and to damage the public's acceptance of state policies. Furthermore, as most of the nation's elite groups and resources are concentrated in the state political apparatus and defense sectors, the availability of human and material resources to be put into actual economic development is bound to be reduced.

Conclusions
This paper uses the periodical Economic Research (one of the representative pieces of primary literature published in North Korea) as the basic text to conduct an in-depth analysis of North Korea's economic policy and its changes over time. Based on the titles and keywords of the articles published from 1980 to 2018, a text mining and machine learning approach was employed to track changes in economic policy after a change of the national leader and in the economic environment.
We find that the title of the paper can be classified into the period of a leader with relatively high accuracy. In addition, the distribution of the thesis topic, along with the scope or direction, varies greatly from one period to another. Since the North Korean economy is inevitably influenced by a single leader, known as the Supreme Leader, the papers published in Economic Research can be regarded as reflecting the policy interests of the leader at the time. By extracting the keywords based on the characteristics of the titles, we gained an understanding of the trends in each leader's policy interests.
The implications of the analysis are as follows: Kim Jong-un is pursuing a different economic policy from those his predecessors pursued through reform and openness. As shown by the New Strategy, declared in 2016, the Kim Jong-un regime envisions a new economic strategy for economic development in the face of economic sanctions, including the knowledge economy, the industrial revolution in the new century, streamlining corporate management, and entering the international trade market. Specifically, the regime fosters the entry of international financial and commodity markets, and the exchange of advanced technologies and knowledge for competitive product development and commercialization. However, economic sanctions resulting from the nuclear weapons development act as fundamental constraints on the country. Nevertheless, improving U.S.-North Korea relations, and easing national security threats are expected to increase access to capital markets and overseas commodity markets; thus, the possibility of full-scale developmental cooperation cannot be excluded. Construction of infrastructure for the inflow of foreign capital, the standardization of financial markets, and general advancement will be needed in line with the reform and opening of North Korea. South Korea, which has geographical, cultural, and linguistic advantages, is in desperate need of cooperation with North Korea, and can take advantage of these positive aspects better than other neighboring countries can; in fact, South Korea may become a fund-raising hub, modeled after Hong Kong, to drive economic growth. It is time to develop realistic policy alternatives to lead a sound market economy with a deeper attention to the economic strategy and policy changes in North Korea.
In this study, we find that many articles appearing in Economic Research during the Kim Jong-un period refer to efficient resource input, corporate competitiveness enhancement, and expert utilization in management, which are emphasized in capitalist market economies. Nevertheless, North Korea still stresses the need to make itself more economically self-reliant to officially achieve its national goal of an independent socialist economy. In the absence of authority, acceptance, and ability, the gap between the state's political rhetoric and the actual economic reality is expected to remain wide.
Compared to the previous two Kim regimes, Kim Jong-un boldly admits to economic failures since his inauguration. It will seek to explore the possibility of economic development through the 'New Five-Year Plan for Economic Development' by 2021. Nevertheless, both 'reforms within the system' and 'reforms of the system', which is essential to North Korea's economic development, would not be an easy choice for Kim Jong-un. A wise alternative should be proposed so that the North Korean economy can go beyond 'muddling-through' and pursue sustainable development with international cooperation.
For logistic regression, the L2 regularization, which regulates the degree of overfitting, is used, and the model is selected by the corresponding hyperparameter. In other words, in minimizing the following cost functions (λ), the number and size of β j estimated according to the size of hyperparameter c are estimated differently, and the cross-validation values of different models are compared to determine the degree of overfitting and underfitting.
The evaluation of the selected model mainly uses recall, accuracy, precision, and F1 score, which are most widely used as evaluation metrics. Table A1 shows the confusion matrix, which shows the combination of four coordinates according to the actual values in the classification and the prediction of the model. TP(TN) is the abbreviation for True Positive (Negative), representing when the model correctly predicts the true value of the feature. If the actual value is positive, or the model's prediction is the opposite, then FN(FP) is used. The following evaluation metrics are calculated by the values of TP, TN, FP, and FN in the confusion matrix. First, accuracy shows the proportion of the actual value and the prediction of the model among the total values. It is used as a rating indicator even in multidimensional classifications, so it is most widely used among the model evaluation indices. Accuracy = TP + TN TP + TN + FP + FN (A3) Unlike accuracy, precision is the proportion of the actual value that the model predicts as positive.
Recall is the ratio of what is predicted by the model as positive among those whose actual value is positive.