Three Commonly Utilized Scholarly Databases and a Social Network Site Provide Di ﬀ erent, But Related, Metrics of Pharmacy Faculty Publication

: Scholarly productivity is a critical component of pharmacy faculty e ﬀ ort and is used for promotion and tenure decisions. Several databases are available to measure scholarly productivity; however, comparisons amongst these databases are lacking for pharmacy faculty. The objective of this work was to compare scholarly metrics from three commonly utilized databases and a social networking site focused on data from research-intensive colleges of pharmacy and to identify factors associated with database di ﬀ erences. Scholarly metrics were obtained from Scopus, Web of Science, Google Scholar, and ResearchGate for faculty from research-intensive (Carnegie Rated R1, R2, or special focus) United States pharmacy schools with at least two million USD in funding from the National Institutes of Health. Metrics were compared and correlations were performed. Regression analyses were utilized to identify factors associated with database di ﬀ erences. Signiﬁcant di ﬀ erences in scholarly metric values were observed between databases despite the high correlations, suggestive of systematic variation in database reporting. Time since ﬁrst publication was the most common factor that was associated with database di ﬀ erences. Google Scholar tended to have higher metrics than all other databases, while Web of Science had lower metrics relative to other databases. Di ﬀ erences in reported metrics between databases are apparent, which may be attributable to the time since ﬁrst publication and database coverage of pharmacy-speciﬁc journals. These di ﬀ erences should be considered by faculty, reviewers, and administrative sta ﬀ when evaluating scholarly performance.


Introduction
One of the activities of faculty within research-intensive colleges of pharmacy is to add to the scientific record through research and scholarly activities. Scholarly metrics, or bibliometrics, generally refers to the analysis of individual productivity and impact through publication and citation counts [1]. Broadly, methods of research performance and impact evaluation for the purposes of promotion, merit, and tenure have been offered that include the Hirsch Index (H-index), journal impact factor, number of publications, authorship position on publication, and citation rates [2][3][4]. Such metrics are used as "judgment devices" by reviewers and the scientific community in evaluation of an individual researcher and the impact of their work [4]. Within schools of pharmacy, these same scholarly metrics are utilized in addition to service, teaching, and clinical activities in the review of faculty performance and promotion [5][6][7]. Previously, obtaining such measures required specialized software or was Faculty names from the included U.S. colleges of pharmacy were collected from the American Association of Colleges of Pharmacy (AACP) and individual college of pharmacy website directories according to our previous published methodology [8]. To be included in the present analysis, U.S. colleges of pharmacy must have had greater than 2 million USD in 2018 U.S. National Institutes of Health (NIH) funding according to AACP grant data and a Basic Carnegie Classification of R1 (very high), R2 (high), or special focus. We chose this cutoff since we wanted to analyze differences from a large, representative sample of research-intensive, NIH-funded U.S. colleges of pharmacy. The Carnegie classification utilizes several criteria to determine research intensity; "special focus" schools were included, since the number-one ranked pharmacy school is special focus, and we did not want to exclude research-intensive schools with this classification [9]. NIH funding was chosen due to its transparent reporting (as compared to foundation or industry funding, which is not readily available), and the median cutoff was used due to the skewed nature of funding across pharmacy schools. Although other definitions have been utilized to define research-intensive colleges of pharmacy, the inclusion of such a large sample size here may assist in minimizing these definitional differences [10][11][12]. Full-time academic faculty with rank were included in this analysis, while adjunct, research, and emeritus faculty were excluded. For each included faculty member, the following data were collected: (1) Name (for database searching), (2) position with rank, (3) department, and (4) university affiliation. Departments were categorized as "clinical" (e.g., Pharmacy Practice, etc.) or "basic" (e.g., Pharmaceutical Science, etc.).
Searches for scholarly metrics were performed in Scopus, Web of Science (WOS), Google Scholar, and the social networking site ResearchGate. Scopus, Web of Science, and Google Scholar were included because they each have wide coverage in biomedical sciences and pharmacy in particular [13][14][15][16]. Although each has a wide coverage, Google Scholar potentially covers more non-traditional sources, such as conference abstracts and graduate theses, which could influence reported metrics [17,18]. Additionally, differences do exist for pharmacy coverage. For example, Mendes and colleagues found that, of 285 identified pharmacy journals, 90% were found in Scopus, while 44.6% were found in Web of Science [19]. Additionally, on a qualitative level, citation counts from these three databases (or some combination thereof) are commonly reported (and within our university, at least one is required) on faculty promotion and tenure applications. Although ResearchGate, a social networking site, should not be considered a primary source for scholarly metric data, we chose to include this database given that it is a highly used social research network site with over 15 million users [20]. This means that, often, researchers are viewing other researchers, their work, and scholarly metrics through this database, making it a potential future model of combining standard scholarly metrics with proposed "altmetrics" [21,22]. However, other points discussed below should be considered for ResearchGate. For Scopus, Web of Science, and Google Scholar, the "author search" utility was used, and for ResearchGate, the general query search bar was used. The faculty member's first, last, and middle initial were input along with their university affiliation (searches performed by B.H., S.F., and A.K.). If no record result was produced, a secondary author (K.B.) performed a verification search. Alternative searches were also performed with additional information obtained from a faculty directory profile, such as past universities, alternate names, or listed publications. An additional, randomized confirmatory search process (K.B.) of records was used to confirm the presence or absence of database records and the accuracy of extracted data. For each database, the following information was collected: Number of documents, H-index, total citations, and highest-cited articles. For Scopus, the year of first publication was collected to calculate the number of publishing years, as this information was readily obtainable from searches. For ResearchGate, the highest-cited article metric was not readily available for collection. Data for each faculty name were collected from their first publication up to June 2019, when the database searching was completed. This research utilizes non-human subject informational databases and, therefore, was exempt from ethics review. The primary research question to be answered was: "What are the differences in scholarly metric reporting amongst major databases, and can any individual or college-level factors predict identified differences?" We believe that individual or college-level factors may have effects on any potential database differences, since they may expose the underlying nature of these databases. For example, clinical faculty may more regularly publish in pharmacy journals that are only covered in Scopus or Google Scholar; therefore, this distinction may be helpful in explaining database differences.
Due to the skewed nature of scholarly metrics, we report the data with both means ± standard deviation and medians with interquartile ranges (IQR), and we compared each common metric between databases using Wilcoxon Signed Rank tests or Spearman Rho correlations [23]. For identification of predictive variables, first, student t-tests were performed to identify significant associations between a given variable and a difference in metrics between databases. Student t-tests were used here, as the calculated differences in metrics (e.g., difference in number of documents between Scopus and Web of Science, etc.) were normally distributed. Since 42 tests were performed for variable identification, a Bonferroni cutoff of 0.001 was utilized. Next, significant variables were entered into regressions. Specifically, linear regressions were performed where the calculated database difference variable (e.g., difference in number of documents between Scopus and WOS, etc.) was the dependent variable, and significant variables from the preceding step were entered as independent variables. Beta estimates and p-values were determined from regression analyses. A correction for multiple comparisons was made for the primary statistics in the manuscript (i.e., differences between databases, correlations between databases, and identification of predictors through regression analyses) by applying a Bonferroni significance level of <0.0004 (117 total tests). All analyses were performed with JMP statistical software version 14.0.

Results
A total of 3023 full-time faculty from 48 U.S. colleges of pharmacy were included (Table S1), which yielded 7713 records from the four searched databases. Table 1 presents an overview of the included faculty. Of note, the number of available Google Scholar and ResearchGate records was lower compared to Scopus and WOS.  Table 2 presents data for common scholarly metrics across the dataset and the percentage difference between them for each metric. All pairwise comparisons were significantly different.

Correlations between Scopus, Web of Science, Google Scholar, and ResearchGate
We performed correlations as another step to gauge metric agreements between the four included datasets ( Table 3). The correlations between the databases for documents were above 0.89, and for H-index, the total citations and highest cited document were all above 0.9. Expanded correlations are found in Supplementary Table S2.

Factors Associated with Metric Differences
To identify individual and college-level factors that are potentially associated with database differences, we first performed student t-tests with available variables of interest to identify significant correlations with database differences. Next, these significant variables were entered into a regression model with the database metric as the dependent variable. Table 4 depicts the best estimates of these regressions. --indicates that the variable was not included in the model. Department type is made up of two levels (clinical and basic) with basic serving as the reference category. Therefore, numbers in this row are clinical relative to the basic science department. Academic Rank is made up of three levels (assistant, associate, and full professor), with full professor serving as the reference category in the regression. Thus, the values for the assistant and associate professor variables are relative to the professor level within the database difference column. A positive value represents an increase in the assistant or associate level relative to the professor level for a given scholarly metric. SC to RG are not shown because only years since first publication were significantly associated with the H-index difference (beta = 0.0440, p = 0.0002). WOS to RG are not shown because only faculty rank and years since first publication were associated with total document difference; assistant professor (beta = 7.18, p = 0.0007), associate professor (beta = 4.74, p = 0.0037), and years since first publication (beta = −0.0813, p = 0.547).

Discussion
The study provides the first comparison of scholarly metrics from highly utilized, easily accessible databases and a social networking site, using data from research-intensive U.S. colleges of pharmacy.
The data indicate that, despite high correlations in their measures, these databases have clear differences in their metric measures which, depending on the metric, is attributable to time or experience-based factors (i.e., faculty rank and years since first publication). The data provided here contribute to the current bibliometric analysis literature by offering insight into the possible differences and similarities between major bibliometric databases in addition to a popular research social network's metrics. To our knowledge, this is the first description comparing the social networking site ResearchGate to other accepted databases; however, other work has similar correlations and differences when comparing Google Scholar, Scopus, and Web of Science in metrics such as H-index, citation growth rates, and number of citations for each paper per author [24][25][26][27]. Similarly to the findings described here, these previous studies have found that Google Scholar provides higher estimates of citation and publication counts, while Web of Science tends to produce significantly lower citation counts [18,24,28]. The approaches in these previous studies were different in that they utilized smaller sample sizes, random selections of researchers across disciplines, or specific journals of research papers for comparisons between databases. Our method of including a large sample size of pharmacy researchers potentially reduces the bias of research field type and specialty. Ultimately, our work adds to this body of literature by aiming to understand how database coverage and algorithms can potentially influence bibliometric estimations.
The Spearman Rho correlations for corresponding metrics between Scopus, Web of Science, Google Scholar, and ResearchGate were all above 0.89, suggesting a high degree of similarity in scholarly metric measurement despite statistically significant differences when tested by the Mann Whitney U. Indeed, the correlations were also significant at the Bonferroni cut-off level of 0.0004. On a qualitative level, Google Scholar had much higher metrics compared to all other databases. Scopus and ResearchGate tended to have estimates with the smallest differences, while Web of Science tended to have the lowest estimates compared to the other databases. These observed differences, yet high correlations, could be due to the large database used here, with over 3000 faculty members and over 7000 database records amongst the faculty members available for analysis. Such differences may be important factors for consideration during faculty evaluations through peer assessment or administrative reviews.
In our analyses to identify potential individual or college-level explanatory variables that could account for the observed differences between databases, it was evident that total publishing years and faculty rank had significant effects on the observed differences. For example, for the difference in documents between Scopus and Web of Science, each year of publication produced approximately 0.4 more documents in Scopus compared to Web of Science. This suggests that as a faculty's total publishing time grows, more documents are captured in the Scopus database compared to the WOS database. Less often, faculty rank had effects on database differences in a pattern similar to that observed with total publishing years. The effects of time and/or seniority factors on database difference may be expected, as they may reflect a wider reach of research and collaborative efforts often observed with senior researchers. Another study, evaluating citation differences in three top journals based on database, identified group authorship (authorship by aggregated groups, such as Consolidated Standards of Reporting Trials (CONSORT)) as the only factor in their analyses significantly associated with the citation differences between databases [29].
It is possible that systematic differences, including content coverage and sources for each database, may account for many of the differences described in Table 2 above [15,30]. Scopus covers over 22,800 titles with 69 million records, while Web of Science's default "Core Collection" includes greater than 21,177 titles with 74 million records [31,32]. This coverage difference is relevant to pharmacy, where certain journals, such as Pharmacy Times, are included in Scopus, but not Web of Science. It may be possible that this is reflected in the data presented here, where all metrics were at least 8% greater in Scopus relative to Web of Science. It may better represent the scholarly productivity of pharmacy faculty, especially clinical faculty that may publish in such pharmacy-based journals, to utilize Scopus metrics in self-evaluations, yearly performance reviews, and promotion and tenure. Interestingly, our multivariate analyses did not identify a significant effect of department type (clinical versus basic) on measured differences, possibly contradicting these hypotheses. It should be noted that the total citations metric was significantly associated with department type (non-significant factor in multivariate analyses), and the total documents metric was non-significantly associated with department type due to the Bonferonni cutoff (p = 0.001).
Google Scholar's coverage is not fully known, but it is estimated they have at least 369 million records and may better capture international works, which likely explains the higher metrics reported here [30,33]. Additionally, Google Scholar's metrics include documents like abstracts and thesis/dissertations, which may not be covered as well in other databases. Content coverage by ResearchGate is not readily available, and their website states that they "import citation data from different sources". To this end, the transparency of ResearchGate as a site for scholarly metric data is lacking. Often, social networking sites aim to incorporate as many users as possible to increase advertising revenue, potentially making their scholarly metric accuracy secondary to other goals. Additionally, ResearchGate has promoted some of their own metrics, such as "Reads" and their "RG score", which have not been validated, found to have little correlation to established metrics but highly correlation to user activity on the site [34][35][36][37]. Of note, our study only utilized total documents, H-index, and total citations in our analyses, which had strong correlations to Scopus metrics (Spearman Rho > 0.8). Finally, ResearchGate's activities, including methods of sharing full-text articles, have been denounced amongst researchers and cited as violating copyright laws [38,39]. In our opinion, despite the high correlations observed in our study, ResearchGate should not be used as a primary source for scholarly metric data without future transparency and assessments.
From an expanded correlational analysis of all metrics between all databases (Table S2), it is evident that lower correlations are observed between publication number and total citations or highest-cited article. This may suggest that the impact of scholarly activity (by citations alone) is not completely correlated with and not fully dependent on the number of publications by a given faculty member [40]. This consideration has been explored by others with proposed alternatives to explore scholarly impact, such as combined article-level metrics, web-based metrics, social-media-based metrics, and comprehensive approaches [41][42][43]. Future work should identify the predictors of total citations or highly cited articles amongst college of pharmacy faculty along with the effect of the database. Such work would also be useful in training junior faculty in approaches to creating high-impact scholarly products.

Limitations and Conclusions
A few limitations should be considered for this study. First, Google Scholar and ResearchGate are self-created profiles compared to Scopus and Web of Science. This is likely the primary reason for the lower number of Google Scholar and ResearchGate records available for analysis. There were many authors with records composed of a high number of documents in Scopus and Web of Science that did not return a profile in Google Scholar and/or ResearchGate, demonstrating the self-creation and self-curation required in these two later databases. The high number of overall records allows us to perform statistical testing between databases; however, is it possible that our results are biased toward faculty that are more likely to self-create these profiles (e.g., faculty with high scholarly productivity, etc.). Additionally, we did not define criteria for including a database. Based on our experience, Web of Science, Scopus, and Google Scholar are now commonly reported on faculty of pharmacy promotion and tenure packets. ResearchGate is not reported as an official metric source; however, we thought that it was useful to include statistics from this social media site, given that many researchers are now using it to "look up" one another, including the metrics in a profile. Second, although our method of collecting faculty profiles was comprehensive, this does not rule out missed or new faculty. To that end, this analysis should be considered a snapshot of the scholarly metrics, and future updates may be needed. The analyses also included covariates that may have a relationship (e.g., time since first publication and faculty rank); there may be other covariates not captured in this study, such as degree of collaboration and specialization or primary field of study, that could influence the results. Although these data are useful for faculty and administrative personnel in colleges of pharmacy, it should be noted that our analysis was restricted to research-intensive U.S. colleges of pharmacy based on NIH funding of wo million or more USD. Thus, these analyses and data may not represent faculty members from non-research-intensive schools. Due to the time-intensive nature required to capture multiple databases on such a large profile of faculty from colleges of pharmacy, our snapshot of 2019 data does not represent the everchanging landscape of metric data. Therefore, future updates may be needed as database coverage is expanded or altered and the faculty makeup of colleges of pharmacy evolves over time.
The data presented here found differences between scholarly metric databases and a social media site despite high overall correlations. Many of these differences can be at least partly attributed to time since first publication by a faculty member; furthermore, they suggest systematic reporting differences between databases. This study provides objective real-world data on which faculty, administrators, and promotion peer-reviewers can understand possible database differences in reporting of scholarly accomplishments. In our opinion, this study does not suggest that one database is more useful or accurate over another, but may suggest that differences exist and that certain databases may provide superior coverage for particular research areas (i.e., clinical pharmacy journals). Future work should consider comparisons with newer transparent databases, such as Dimensions from Digital Science, which further incorporates grant data with standard scholarly metric data. This will assist in validating and comparing the ever-expanding landscape of citation databases.