Scientific publications have been, and continue to be, the main vehicle for the dissemination of science. Its study and evaluation therefore allow the study of data of great value that, applied to countries, institutions, geographic communities, etc., make it possible to apply greater objectivity to decision-making in scientific policy [1
]. In developed countries, increasing importance is given to the evaluation of the scientific production and the international visibility of the scientific works of a country; it is an indicator widely used as a reference of the quality of the research carried out [2
]. The number and quality of publications in scientific journals are one of the criteria for evaluating the research and teaching suitability of a professor, a research group, a department or a university.
In knowledge management, transferring knowledge is becoming increasingly important, mainly through publication in different sources of information that facilitate its dissemination among the scientific community. In the specific case of Psychology, scientific articles are the main source of knowledge transmission. These works published in scientific journals are in turn collected in different bibliographic databases, such as the Web of Science (WoS) or Scopus, which classify the scientific journals that they index in different rankings. One of the most used and controversial indicators of the journals is the one known as “Impact Factor” (IF), which is included in the Journal Citation Reports (JCR) of the WoS. The IF was designed to measure the influence of scientific journals by counting the number of citations that they have obtained (in a period of two years), prior to the year in which the recount is made. Although this indicator has now established itself as a benchmark for quality, it is not without controversy [3
], since publishing in a journal with a high Impact Factor does not imply that the article in question is of quality, or even that it has received many citations, but that the set of articles published over two years in that journal have been widely cited. However, in the case of the Spanish scientific scene, it is the journals that have a high Impact Factor which enjoy a greater weight when assessing the scientific careers of researchers, with special relevance to whether these works are published in Q1 or Q2 journals, as envisaged by the National Agency for Evaluation and Prospective (ANEP
in Spanish: Agencia Nacional de Evaluación y Prospectiva
), the National Commission for the Evaluation of Research Activities (CNEAI
in Spanish: Comisión Nacional Evaluadora de la Actividad Investigadora
) and the National Agency for the Evaluation of Quality and Accreditation (ANECA
in Spanish: Agencia Nacional de Evaluación de Calidad y Acreditación
], if we look at the quality indicator of the ANECA, especially in the case of Health Sciences and Social Sciences, as is the case of Psychology.
One issue that generates controversy is related to the criteria used to evaluate research productivity, and here different positions appear. Hiwever, the most polarized are those that defend and those that criticize the use of bibliometric indicators to evaluate academic performance [5
]. In some countries, such as Spain, the implementation of an evaluation system for research activity plays a crucial role in the international dissemination of scientific production, but it also has some negative aspects, since it limits research worthy of being evaluated positively to that published in English journals, those included in the WoS databases, which harms research in social and humanistic sciences [6
], for Spanish researchers must publish in journals of greater impact and in English to improve the scientific production indexes of Spanish universities, but this should not mean that this is the only production that should be considered for the purposes of evaluation or that Spanish or Ibero-American journals do not publish articles of similar quality [7
]. However, over and above the obsession with the Impact Factor or the English language, it should be borne in mind that the most appropriate journal to publish a document should depend on the audience to which it is addressed [8
Another question that has aroused controversy with regard to the evaluation of scientific quality through scientific publications has been the control of these by editorial groups. According to some experts [9
] there are five main publishing groups: Elsevier (Holland), Springer (USA), Wiley-Blackwell (USA), Taylor-Francis (United Kingdom) and Sage (USA). These publishing companies have the largest oligopoly part of the publications in the WoS, editing 70% of the journals indexed in the WoS in Social Sciences. Faced with this oligopoly, as well as the methodology used through bibliometric indicators to evaluate the quality of the works, the publications or the scientific careers of the research, the San Francisco Declaration on the evaluation of research [10
] and the Leiden Manifesto [11
] includes a series of recommendations, all of which are relevant, among which the importance of local and national research is worth mentioning, since there are many research of scientific quality of a local or national nature that are relevant to a specific community and present serious limitations at the time of publication.
On the other hand, most of the scientific contributions in different types of documents are collected in databases, one of the main ones being the WoS of the Clarivate Analytics company in the United States of America, where most of the publications are in English. This can generate biases when assessing works mainly of a local or national nature, as pointed out by [11
]. In the case of scientific journals, and specifically journals in the area of Psychology, they are included in the Science Citation Index (SCI) databases in the Psychology category, and the Social Sciences Citation Index (SSCI) database, in the thematic categories of “Applied”, “Biological”, ”Clinical”, “Developmental”, “Educational”, “Experimental”, “Mathematical”, “Multidisciplinary”, “Social”, and “Psychoanalysis”, annually publishing an evaluation of the journals included in the SCI and SSCI categories according to the number of citations in the well-known Journal Citation Reports (JCR).
In the case of journals classified in the Psychology categories of the 2017 JCR (SCI and SSCI), these were published by a total of 30 countries: Argentina, Australia, Belgium, Brazil, Canada, Chile, China, Colombia, Czech Republic, Denmark, France, Germany, Italy, Japan, Mexico, Netherlands, New Zealand, Poland, Portugal, Romania, Russia, Serbia, Slovakia, South Africa, Spain, Sweden, Switzerland, Turkey, United Kingdom and United States of America. From 2014 to 2016, approximately 80% of the journals included in the JCR (SCI and SSCI) were published by the United States of America and England, followed by those published in the Netherlands, Germany and Spain. This low representation of journals published by other countries is more accentuated when the number of countries with journals in the first quartiles is counted. Specifically, eight countries had representation in these years in Quartile 1 and 12 in Quartile 2 [12
The publication language is related to the position of the journals in the JCR. As pointed out by some experts [13
], the scientific journals better positioned in the JCR are published in English and have higher Impact Factors. Even more, it is not difficult to find Ibero-American journals publish their works only in English [16
]. Researchers [17
] find that in the case of multilingual Spanish biomedical journals, articles in English receive a higher number of citations compared to articles published in Spanish, although these works are mainly signed and cited by foreign authors. Albeit the publication in a language other than English does not imply that the article does not have quality, as noted by [18
], in the case of works published in Spanish.
As pointed out by [20
], the presence of journals published in Spanish-speaking countries in the 2015 JCR represents only 1.8% of the total number of journals in all the thematic categories, and of these, only 0.8% published articles in Spanish. Of the 152 journals analyzed, only 2% were in Quartile 1, 12.5% in Quartile 2, 15.1% in Quartile 3, and 70.4% in Quartile 4.
After describing part of the scenario related to scientific research, such as editorials, scientific journals, databases where journals are collected, specifically the JCR bases (SCI and SSCI) of the WoS, and the countries where the journals are published, the present work has the objective to compare the evolution of Psychology journals in the JCR (SCI and SSCI) in the last 20 years, from 1998 to 2017, published in Ibero-American countries and in the Netherlands with a view to know the main differential characteristics between both, which may be conditioning their evolution and position in the JCR.
As specific objectives, it was proposed to know the characteristics of Ibero-American and Dutch Psychology journals in terms of journal editorial, categories in which they are classified in the JCR, open-access publication, publication language, published numbers per year, origin of the main contributions, quartiles, deciles and position reached within the thematic area of Psychology in which they are classified. The starting hypothesis was that there would be differential characteristics between Ibero-American and Dutch Psychology publications, which would be behind the position occupied by the journals in the JCR (SCI and SSCI), since these databases are mainly characterized by collecting mainly journals published in the United Kingdom and the United States of America, with a predominance of works published in the English language and with a greater number of journals published by large publishing groups compared to publications published by foundations or universities.
Finally, based on the hypotheses proposed, it is intended to offer a reflection on how national policies of scientific evaluation can contribute to the invisibility of other sources of information, such as in the case of Ibero-American Psychology journals, indexed in other bases or platforms such as SciELO or Latindex in the Ibero-American context, or Scopus.
3. Materials and Methods
A total of 62 journals were analyzed, of which 25 were published in Ibero-American countries and 37 were published in the Netherlands. Ibero-American journals and the Dutch journals were classified into 9 Psychology thematic areas, with the exception that among Ibero-American journals, there is one journal in the “Psychoanalysis” category and none among the Dutch journals; although there are several Dutch journals in the “Psychology Biological” category and no Ibero-America journals in that category. In Table A2
and Table A4
, you can notice the thematic areas in Psychology.
The number of analyzed records of Ibero-American Psychology journals was 243 and 702 of Dutch Psychology journals, a total of 945 records having been analyzed. In this work, registration is understood as the number of times a journal appears in the JCR, either in SCI or in SSCI, in the period from 1998 to 2017.
The Ibero-American journals selected in the present study were the following:
Edited in Argentina: Revista Argentina de Clínica Psicológica and Revista Iberoamericana de Diagnóstico y Evaluación e-Avaliaçao Psicologica.
In Brazil: Psicologia Reflexao e Critica; Revista Latinoamericana de Psicopatología Fundamental and Tempo Psicoanalítico.
In Chile: Terapia Psicológica.
In Colombia: Universitas Psychologica and Revista Latinoamericana de Psicología.
In Mexico: Revista Mexicana de Psicología.
In Portugal: European Journal of Psychology of Education.
In Spain: International Journal of Clinical and Health Psychology; European Journal of Psychology Applied to Legal Context; Psicothema; Revista de Psicodidáctica; Behavioral Psychology–Revista de Psicología Conductual; Anales de Psicología; Revista de Psicología del Deporte; Spanish Journal of Psychology; Psicológica; Psychosocial Intervention; Journal of Work and Organizational Psychology–Revista de Psicología del Trabajo y de las Organizaciones and Clínica y Salud.
In the case of the journals Estudios de Psicología, Revista de Psicología Social and Infancia y Aprendizaje, these are no longer published in Spain since 2012 and are published in the United Kingdom.
The journals published in the Netherlands were: Journal of the American Academy of Child and Adolescent Psychiatry; Cognition; Biological Psychology; International Journal of Behavioral Development; International Journal of Psychophysiology; Journal of Happiness Studies; Mindfulness; Body Image; Sexual Abuse—A Journal of Research Treatment; Journal of Applied Research in Memory and Cognition; Psychology of Sport and Exercise; Eating Behaviors; Acta Psychologica; European Journal of Social Psychology; Multisensory Research; Archive for the Psychology of Religion; Gedrag and Organisatie; International Journal for Educational and Vocacional Guidance; Studies in Educational Evaluation; Social Justice Research; Social Psychology of Education; Cognitive Systems Researchs; Journal of Economic Psychology; Reading and Writing; Clinical Neuropsychologist; Journal of Psychosomatic Obstetrics and Gynecology; Learning and Individual Differences; Instructional Science; Behavioural Processes; Aging Neuropsychology and Cognition; Human Movement Science; Journal of Clinical and Experimental Neuropsychology; Spatial Vision; Applied and Preventive Psychology; Seeing and Perceiving; Psychologie and Gezondheid and Journal of Contextual Behavioral Science.
Regarding the procedure, all the Psychology journals collected in the JCR (SSCI and SCI) edited in Ibero-American countries and in the Netherlands from 1998 to 2017 were selected. For this, the JCRs of each of the years were consulted. The choice of journals published in the Netherlands for this study compared to those of other countries was due to several factors: on the one hand, those that were not published in an English-speaking country, discarding the journals published in the UK, USA, Canada, Australia and New Zealand. The second criterion used was that they were published in a European country, given the greater presence of Spanish journals in the Ibero-American journal sample. Finally, they should count with a number of journals as similar as possible to the sample of Ibero-American journals, taking into account the number of journals per country in the JCR in recent years [12
]. This reduced the choice to journals published in two countries, Netherlands and Germany. Finally, it was decided to discard the German journals because some of them were published in German and English, compared to the Dutch ones that were published in English, this language being the second most used language in Ibero-American Psychology journals.
The consultation of Ibero-American journals was carried out by individually selecting each Ibero-American country that appeared in the JCR and that had had a Psychology journal indexed in the JCR in the period studied, being the countries of: Argentina, Brazil, Chile, Colombia, Mexico, Portugal and Spain.
After the selection of the journals, the variables selected for the study were consulted in the JCR. Considering the position and the number of journals that year in the category, we proceeded to calculate the decile in order to acquire more specific information about the position of the journal. In addition, and with the aim of knowing the evolution of Ibero-American and Dutch journals according to the quartiles, we proceeded to calculate the frequency and percentages by diving the period studied into four periods of 5 years. Subsequently, the data were analyzed, the results obtained, and the conclusions drawn.
The counting of frequencies, means, calculations of deciles and percentages for each of the analyzed journals was carried out. Chi-square tests were carried out to determine the existence of differences between the variables based on journal editions, with the Ibero-American journals and Dutch journals categories, years of indexation in the JCR, including from 1998 to 2017, and the quartile, all through the statistical package SPSS 24.
The results found in the present study show a clear discrepancy between the Psychology journals published in Ibero-American and Dutch journals, differences that may bias their position in the JCR and the evolution of the journals over the years.
The quality of the publications taking into account the number of times and the percentages in which journals are classified in the first quartiles and deciles does not admit any doubt about the best position of the Dutch Psychology journals, since 54.2% of the records analyzed were publications placed between the first quartiles, compared to 18.2% of the records of Ibero-American Psychology journals, percentages very similar to those found by [20
]. These differences are even greater if we look at the decile occupied by the journal in each of the years analyzed, almost a quarter of the Dutch Psychology journals were among the first three deciles and approximately 55% were among the first five deciles, a very different result to the positions occupied by Ibero-American Psychology journals, in which almost 59% of the records were in the last two deciles.
However, when it comes to understanding these results, we must take into account other differential variables between the Psychology journals published in Ibero-America and those published in the Netherlands, which have to do with the scenario, already described in the introduction, about how scientific quality is measured at present. These contextual variables may bias the results with respect to the positioning of the journals in the JCR, and, therefore, evaluate a journal as having better or worse scientific quality solely because of the Impact Factor. As pointed out by [21
], the Impact Factor of the journal where the article is published does not mean that the article has scientific value, there are other bibliometric indicators that can better explain the relevance of the article.
For this and responding to one of the objectives set out in this study, there are great differences between Ibero-American and Dutch Psychology journals. On the one hand, it highlights that the large publishing groups that have the oligopoly of scientific publications [9
] are present in 31 of the Dutch journals (83.8%) versus the scarce presence of these publishers in Ibero-American journals, where local entities such as foundations or universities are mainly the most represented as journal publishers. This circumstance can even explain the greater number of volumes per year published by Dutch Psychology journals compared to Ibero-American journals, to the extent that large publishers can count on greater means when publishing in front of small publishers.
On the other hand, it is worth mentioning the publication language, where in the vast majority of Dutch journals, English is the only language of publication, compared to Ibero-American journals in which Spanish remains the most widely used language, although many of them admit works in different languages. If we take into account the position of the Dutch and Ibero-American journals analyzed, the journals with the best position in the JCR are the ones that publish in English, as pointed out by [12
], a relationship that is also fulfilled in the specific case of Ibero-American journals, where those with the best position publish their works in different languages, mainly English and Spanish, as in the case of the International Journal of Clinical and Health Psychology, European Journal of Psychology to Legal Context or Psicothema. On the other hand, only one Ibero-American journal—Spanish Journal of Psychology—publishes its works in English, a circumstance already pointed out by [15
]. The language of publication in turn is related to the origin of the authors who sign papers in the journals, with a clear tendency to American and English authors in Dutch journals and, in the case of Ibero-American Psychology journals, a greater presence of Ibero-American authors and of English-speaking countries in the case of multilingual journals.
Other differences are, on the one hand, the number of publications and years of permanence in the JCR, which is much higher in the Dutch journals, an aspect that may be related to the greater presence of large publishing groups in Dutch journals. With respect to the number of categories in Psychology and other scientific disciplines, Dutch journals have a greater presence compared to Ibero-American journals, an aspect that may be related to the editorial policies of journals. In this sense, it could be asked whether a greater presence in different categories is related to greater dissemination of the journal and therefore a greater number of citations.
These results only confirm that, currently and according to the historical trajectory, the Ibero-American Psychology journals in the JCR, in the years analyzed, have increased their number over the years. There has not been an improvement in position according to the quartile and the positions they occupy in each and every one of the areas of Psychology, an aspect that may be conditioned by the biases related to the current scientific scenario. In this sense, it becomes relevant to minimize the effects of research biases, the recommendation included in the San Francisco Declaration of the Evaluation of Research [10
] and the Leiden manifesto [11
], which have to be taken into consideration in the research evaluation policies.
The differential characteristics of Ibero-American journals versus Dutch ones, with a greater presence of local publishers versus the large publishing groups that characterize the Dutch journals to a greater extent, as well as the indicators used by the Spanish agencies when evaluating researchers’ careers, leave aside and reduce interest in researchers who study regional or local variables, which can affect even the sustainable development of a region. Also, it was pointed out by [22
] that local development implies, on the one hand, the use of endogenous resources, such as publishers belonging to organizations or universities and, on the other hand, the improvement of the well-being and quality of life of a specific population, which is also achieved with the development of research directed to the study of social or contextual variables of the Social Sciences, which characterize a specific region. The example of journals as sources of knowledge transmission, as in the case of Dutch journals compared to Ibero-American Psychology journals, leaves us to see what they [23
] point out as the danger of globalization towards sustainable development, and that some form also includes the Declaration of San Francisco [10
] and the Leiden Manifesto [11
], by supporting local and national studies and the limitations that these have at the time of being published.
Future studies must address new ways of assessing the scientific quality of articles, journals and researchers, respecting the differences between regions, considering the relevance of local studies and publications, and even differences by thematic categories, generating indicators far from the biases that seem to surround a scientific scenario conditioned by quantity versus quality and by business. These studies should serve to guide the national policies of the scientific evaluation of researchers, taking into account the cultural, political and contextual characteristics of each region or country, as well as the public to which it is addressed, which condition the sources of information to ones that the researchers publish their scientific results. In this sense, as pointed out by [18
] with respect to the language of the publication and not so much in relation to the journal in which the work is published, there are articles of great scientific relevance written in Spanish.
Among the limitations of the present study, it can be mentioned that only the JCR Psychology journals of a country have been chosen for comparison with journals published in Ibero-American countries, as well as the single Psychology journals that have been selected.
It is worth reflecting on how information sources cease to be useful for local or national communities when they are inserted into culturally, socially and politically related databases, as in the case of Ibero-American Psychology journals in the JCR. In the specific case of Spanish science, the evaluation policy of scientific careers has contributed to this, prioritizing the position of the source of information where the work is published rather than the relevance or quality of what is published.
In the case of Ibero-American Psychology journals indexed in the JCR (SCI and SSCI), it is observed that although there is an increase in the number of journals in this database over the years, there is stagnation regarding the position according to the quartile they occupy, an aspect that may be related to very different characteristics such as the language of publication, publication in open access journals or the publishing group, with respect to journals published in other countries such as in the case of Dutch journals. In the specific case of Ibero-American Psychology journals in the JCR, one could speak of the existence of a glass ceiling, since different variables may be conditioning the achievement of higher positions in these databases. These variables are inherent to the specific characteristics of the Ibero-American countries, Portugal and Spain, not only because of the language of publication, mainly Spanish and Portuguese, but also because of the variables related to the journal’s own edition, which falls on publishers with a marked local or national character. This publishing origin must not be seen as negative, but rather it allows responding to demands related to socially and territorially specific groups, given the great cultural and social diversity that the Ibero-American countries present. Proof of this is the creation of two platforms of Ibero-American journals, namely, Latindex (Regional Online Information System for Scientific Journals of Ibero-American countries, the Caribbean, Portugal and Spain) and SciELO (Scientific Electronic Library Online), that allow the collection of a large number of journals published in these countries. All this can lead to Ibero-American scientific publications, in the case of Psychology, being indexed in English language databases with a large number of journals published by enormous publishing groups, as is the case for Dutch Psychology journals in the JCR, occupying lower JCR positions, as quartiles Q3 and Q4. In this sense, the Spanish scientific evaluation policy, which prioritizes the publication of articles in journals positioned in Quartile 1 or 2 [4
], could be contributing to the invisibility of other sources of information in the Social Sciences areas, such as Ibero-American Psychology publications collected in databases such as SciELO or Latindex, or those indexed in Scopus.
It is worth reflecting on the sources of information in the area of social sciences, in the sense that, from this scientific field, it is as relevant to have access to local events as those occurring globally—an aspect that was considered in this scientific context when establishing the evaluation criteria of science and scientists—and, therefore, apply the different recommendations that have been made both in the declaration of San Francisco and in the Leiden manifesto.