Which Are the Most Influential Cited References in Information?

This bibliometric study presents the most influential cited references for papers published in the journal Information by using reference publication year spectroscopy (RPYS). A total of 30,960 references cited in 996 papers in the journal Information, published between 2012 and 2019, were analyzed in this study. In total, 29 peaks with 48 peak papers are presented and discussed. The most influential cited references are related to set theory and machine learning which is consistent with the scope of the journal. A single peak paper was published in the journal Information. Overall, authors publishing in the journal Information have drawn from many different sources (e.g., journal papers, books, book chapters, and conference proceedings).


Introduction
The MDPI journal Information covers a broad range of subject areas: Information theory and methodology (e.g., coding theory, information-theoretic security, quantum information, and philosophy/ethics of information), information systems (e.g., knowledge management, social media/social networks, and big data/cloud computing), information processes (e.g., digital signal processing, data mining, and information extraction), information applications, (e.g., man-machine interface, information in society/social development, and business process management), and information/communication technology (e.g., communication systems/networks, wireless sensor networks, and mobile communication services), see also [1]. The tenth anniversary of a journal is also an opportunity to look back at the papers published so far and which sources (i.e., cited references) were most influential. This study employs reference publication year spectroscopy (RPYS) [2] to achieve this goal.
In the first step of an RPYS analysis, a publication set is gathered. Such a publication set can comprise a scientific field (e.g., Higgs boson research [3], climate change [4], dentistry, and neurosciences [5,6], health equity [7], or density functional theory [8][9][10]), a journal (e.g., the journal Ecological Economics [11] and the journal FEMS Microbiology Letters [12]), or the oeuvre of a researcher (e.g., oeuvre of Eugene Garfield [13]). Reference publication year spectroscopy analysis focuses on the references cited in publications and, therefore, employs a backward view from the published papers, in contrast to a times cited analysis which employs a forward view. In the second step, the number of cited references (NCRs) for each of the reference publication years (RPYs) are plotted in a spectrogram. In the third step, the peaks in the spectrogram are inspected to find the most frequently cited references in addition to ordering the cited references by their NCR values. The peaks point to cited references which have been referenced very frequently in the initial publication set. By inspecting the earliest peaks, the historical roots of the initial publication set can be determined. The meaning of the cited references should be interpreted by an expert in the field.

Materials and Methods
Two bibliographic databases were used for the study: Web of Science (WoS, provided by Clarivate Analytics) and Scopus (provided by Elsevier). The MDPI journal Information is covered in WoS only since 2015 via the emerging sources citation index (ESCI). Scopus indexes the journal since 2012. However, the structure of the cited references is more systematic in WoS than in Scopus. Therefore, Scopus data were used for the time frame 2012-2014, and WoS data were used thereafter. The papers published in the MDPI journal Information were downloaded with cited references information on 29 October 29 2019.
We used the CRExplorer [14][15][16] (see also crexplorer.net) to convert the Scopus download in the WoS format to import the full publication set in CRExplorer. Nine hundred and six publications with 30,960 cited references were imported from the WoS download; 649 cited references without RPY were discarded. Ninety publications with 3551 cited references were imported from the Scopus download. The earliest cited references date back to the 17th century with René Descartes (1637, "Discourse on the Method"), but this reference was cited only once and, therefore, removed (see below). CRExplorer offers a feature to disambiguate cited references. This feature was used to merge equivalent cited references considering volume and page number with the Levenshtein threshold of 0.75. Afterwards, cited references occurring only once were removed to sharpen the spectrogram. Additionally, the cited references in peak years and the cited references occurring at least ten times were merged manually. This procedure led to a set of 2044 cited references within the time frame 1859-2019.
The NCR and the five-year median deviation of the NCR (years x, x − 1, x − 2, x + 1, and x + 2) were plotted. For the identification of peak years (RPYs with higher NCR values than the neighboring years), both curves were inspected. Cited references which were mainly responsible for the peak are referred to as peak papers in the following. The specific level of NCR values differed by publication set, citing year, and cited year. The NCR values should not be compared across RPYs.

Materials and Methods
Two bibliographic databases were used for the study: Web of Science (WoS, provided by Clarivate Analytics) and Scopus (provided by Elsevier). The MDPI journal Information is covered in WoS only since 2015 via the emerging sources citation index (ESCI). Scopus indexes the journal since 2012. However, the structure of the cited references is more systematic in WoS than in Scopus. Therefore, Scopus data were used for the time frame 2012-2014, and WoS data were used thereafter. The papers published in the MDPI journal Information were downloaded with cited references information on 29 October 29 2019.
We used the CRExplorer [14][15][16] (see also crexplorer.net) to convert the Scopus download in the WoS format to import the full publication set in CRExplorer. Nine hundred and six publications with 30,960 cited references were imported from the WoS download; 649 cited references without RPY were discarded. Ninety publications with 3551 cited references were imported from the Scopus download. The earliest cited references date back to the 17th century with René Descartes (1637, "Discourse on the Method"), but this reference was cited only once and, therefore, removed (see below). CRExplorer offers a feature to disambiguate cited references. This feature was used to merge equivalent cited references considering volume and page number with the Levenshtein threshold of 0.75. Afterwards, cited references occurring only once were removed to sharpen the spectrogram. Additionally, the cited references in peak years and the cited references occurring at least ten times were merged manually. This procedure led to a set of 2044 cited references within the time frame 1859-2019.
The NCR and the five-year median deviation of the NCR (years x, x − 1, x − 2, x + 1, and x + 2) were plotted. For the identification of peak years (RPYs with higher NCR values than the neighboring years), both curves were inspected. Cited references which were mainly responsible for the peak are referred to as peak papers in the following. The specific level of NCR values differed by publication set, citing year, and cited year. The NCR values should not be compared across RPYs.   In the following, a selection of cited references is discussed which is mainly responsible for the observed peaks in the spectrogram. The interested reader can obtain the full list of 2044 cited references at the Supplementary Materials. In the following, the RPYS results are divided into two parts: (i) the time frame 1858-1989 and (ii) the time frame 1990-2019.

RPYS Analysis for the Time Frame 1858-1990
The RPYS spectrogram for the time frame 1858-1990 is shown in Figure 2. Twenty-one peaks, some of them very small, can be observed in this time frame: 1859, 1902, 1928, 1935, 1937, 1943, 1946, 1948, 1950, 1954, 1957, 1959/1960, 1965, 1967, 1970, 1972, 1975, 1980, 1984, 1986, and 1989. Information 2019 In the following, a selection of cited references is discussed which is mainly responsible for the observed peaks in the spectrogram. The interested reader can obtain the full list of 2044 cited references at the supplementary material.
In the following, the RPYS results are divided into two parts: (i) the time frame 1858-1989 and (ii) the time frame 1990-2019.

RPYS Analysis for the Time Frame 1858-1990
The RPYS spectrogram for the time frame 1858-1990 is shown in Figure 2. Twenty-one peaks, some of them very small, can be observed in this time frame: 1859, 1902, 1928, 1935, 1937, 1943, 1946, 1948, 1950, 1954, 1957, 1959/1960, 1965, 1967, 1970, 1972, 1975, 1980, 1984, 1986, and 1989. A selection of peak papers which are mainly responsible for the observed peaks in the spectrogram in Figure 2 is shown in Table 1 and shortly discussed below. The earliest peak, and the only peak in the 19th century, was due to four references to Darwin's classic that revolutionized natural science (in particular biology) and altered our understanding of the world (CR1). A selection of peak papers which are mainly responsible for the observed peaks in the spectrogram in Figure 2 is shown in Table 1 and shortly discussed below. The earliest peak, and the only peak in the 19th century, was due to four references to Darwin's classic that revolutionized natural science (in particular biology) and altered our understanding of the world (CR1).
In 1902, the Muirhead mean was introduced in CR2 which is the only cited reference below the first peak in the 20th century. The peak in 1928 was composed of two cited references (CR3 and CR4). In CR3, Hartley developed a quantitative measure of "information" which is based on physical, in contrast to psychological, considerations. In CR4, Bohr contributed general remarks to the discussion on the quantum postulate and the development of the atomic theory with the hope to harmonize the different views on the topic. The peak in 1935 was only due to the fact of three references to CR5 which has become known as the Einstein-Podolsky-Rosen (EPR) paradox. The authors (as most of their colleagues at the time) struggled with the quantum-mechanical description of physical reality.
The peak in 1937 was mainly due to the fact of Turing's contribution (CR6) to the topic of computable numbers (i.e., real numbers which have expressions that are calculable by finite means) with an application to the Hilbertian Entscheidungsproblem ("decision problem"). The small peaks in 1943 and 1946 were composed of a single cited reference each, CR7 and CR8. In CR7, McCulloch and Pitts discussed a logical calculus of the ideas immanent in nervous activity. In CR8, Ryle, Lewy, and Popper suggested that a statement can be logically true (in the case of mathematical inference) but also falsifiable as a statement about the world (in the case of event interpretation and inference). In CR9, Shannon initiated classical information theory. The cited reference CR9 and another less frequently cited reference (therefore not included in Table 1) are the only cited references in the peak year 1948. Although the NCR value of 16 in Table 1 might seem low, CR9 is the sixth most cited reference in this RPYS analysis. In CR10, Turing proposed what is known today as the Turing test. In CR11, Everett introduced what is known today as the many-worlds interpretation of quantum theory. The peak in 1959/1960 was mainly composed of two cited references, CR12 and CR13. The cited reference CR12 is the famous book about the philosophy of science by Popper. In CR13, Cohen proposed a coefficient of agreement for nominal scales. The peak in 1965 was mainly due to the fact of CR14, in which Zadeh proposed fuzzy sets (sets with a gradual assignment of the membership of elements in a set). The most cited reference below the peak in 1967 was CR15. In this conference paper, MacQueen proposed the clustering algorithm k-means.
The peak in 1970 was mainly due to the presence of two quite different cited references, CR16 and CR17. The cited reference CR16 is Kuhn's famous book about the history of science with the distinction between normal science and scientific revolutions. In CR17, Bellman's and Zahdeh's contributed to the topic of fuzzy decision making. The most cited reference below the peak in 1972 was CR18. Here, Anderson discussed the broken symmetry and the nature of the hierarchical structure of science providing some examples thereof. The peak in 1975 was mainly due to the Zahdeh's proposal of the concept of linguistic variables (CR19). The most cited references below the peak in and around 1980 were Saaty's introduction of the analytic hierarchy process as an effective tool for dealing with complex decision making (CR20) and Porter's proposal of an algorithm for automated suffix stripping (CR21).
The remaining two peaks in the time frame 1858-1990 were mainly due to the fact of Atanassov's works on intuitionistic fuzzy sets. In CR22, Atanassov defined the concept of intuitionistic fuzzy sets. In CR23, Atanassov generalized the notion of intuitionistic fuzzy sets in the spirit of ordinary interval-valued fuzzy sets. The rather small peaks in 1954 and 1984 were composed of 5 and 6, respectively, different cited references which are cited only twice each. It would be arbitrary to pick one or a few of them. Also, it would not be useful to discuss all of these eleven cited references here.

RPYS Analysis for the Time Frame 1990-2019
The RPYS spectrogram for the time frame 1990-2019 is shown in Figure 3. Eight peaks, some of them very broad, can be observed in the spectrogram : 1994/1995, 1996-1999, 2000-2002, 2003-2005, 2006/2007, 2010-2012, 2013-2015, and 2016/2017. A selection of peak papers which were mainly responsible for the observed peaks in the spectrogram in Figure 3 is shown in Table 2 and shortly discussed below.    Table 2. Selection of peak papers which is mainly responsible for the observed peaks in the spectrogram in Figure 3.

CR Number RPY Cited Reference NCRs
Information 2019, 10, 395 7 of 9 The first and smallest peak of the time frame 1990-2019 was mainly composed of three cited references which contributed to machine learning (CR24, CR25, and CR26). In CR24, Bengio, Simard, and Frasconi showed why gradient-based learning algorithms face an increasingly difficult problem in the case of long-term dependencies. In CR25, Kennedy and Eberhart introduced a concept for the optimization of non-linear functions using a particle swarm methodology. In CR26, Cortes and Vapnik proposed the popular support vector network model for machine learning.
The peak across RPYs 1996-1999 was mainly composed of four cited references (i.e., CR27, CR28, CR29, and CR30). In CR27, Herrera, Herrera-Viedma, and Verdegay present a consensus model in group decision making under linguistic assessments. In CR28, Hochreiter and Schmidhuber introduced a novel and efficient gradient-based method called long short-term memory. In CR29 and CR30, Smarandache introduced neutrosophic sets as generalization of fuzzy sets, intuitionistic fuzzy sets, and interval-valued intuitionistic fuzzy sets.
Mainly, three cited references (i.e., CR31, CR32, and CR33) were responsible for the peak across RPYs 2001-2003. In CR31, Breiman introduced the popular random forest methodology. In CR32, Lafferty, McCallum, and Pereira proposed the conditional random field model. In CR33, Blei, Ng, and Jordan described the latent Dirichlet allocation (LDA), a generative statistical model in natural language processing.
The cited references which were mainly responsible for the peak 2004-2006 were CR34, CR35, CR36, CR37, CR38, and CR39. In CR34, Lowe presented a method for extracting scale-and rotation-invariant features from images that can be used to perform reliable matching between different views of an object or scene. In CR35, Wang, Bovik, Sheikh, and Simoncelli introduced an alternative complementary framework for image quality assessment based on the degradation of structural information. In CR36, Xu proposed the concept of an uncertain linguistic variable. In CR37, Wang, Smarandache, Zhang, and Sunderraman introduced interval neutrosophic sets. In CR38, Demšar proposed the sign test. In CR39, Bishop introduced the cluster validity index which is capable of providing a quality measurement for the goodness of a clustering result for a data set.
With the peak 2010-2012, we enter the decade in which Information has been publishing so far. This peak was mainly composed of three cited references (i.e., CR40, CR41, and CR42), one of them being published in Information (CR42). In CR40, Wang, Smarandache, Zhang, and Sunderraman introduced single-valued neutrosophic sets. In CR41, Collobert and coworkers proposed a unified neural network architecture and learning algorithm for natural language processing. In CR42, Logan reviews the historic development of the concept of information including the relationship of Shannon information and entropy.
The peak 2013-2015 was mainly composed of five cited references (i.e., CR43, CR44, CR45, CR47, and CR47). Mikolov and co-workers proposed the popular Doc2vec methodology in CR43. In CR44, Ye presented the correlation and correlation coefficient of single-valued neutrosophic sets based on the extension of the correlation of intuitionistic fuzzy sets. In CR45, Pennington, Socher, and Manning proposed the GloVe method (global vectors for word representation). In CR46, Ye proposed a multi-criteria decision-making method using aggregation operators for simplified neutrosophic sets. In CR47, LeCun, Bengio, and Hinton provided a review about deep learning.
Finally, many cited references contributed to the peak 2016/2017. The most cited reference was CR48. Here, Krizhevsky, Sutskever, and Hinton presented large, deep convolutional neural networks to classify high-resolution images.

Discussion and Conclusions
Compared to other RPYS analyses of publication sets of a similar size (see for example Reference [17]), individual cited references of the current study have rather low NCR values. Instead of exhibiting large NCR values, more important cited references were found. This seems to reflect a rather broad basis on which the authors publishing in the journal Information have drawn on.