Next Article in Journal
The Research Environment in a Developing Economy: Reforms, Patterns, and Challenges in Kazakhstan
Previous Article in Journal
The Influence of International Collaboration on the Scientific Impact in V4 Countries
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Publications
Volume 10
Issue 4
10.3390/publications10040036
publications-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference

by
Nicholas Yee Liang Hing
1,*,
Xin Ci Wong
2,
Pei Xuan Kuan
2,
Mohan Dass Pathmanathan
2,
Mohd Aizuddin Abdul Rahman
2 and
Kalaiarasu M. Peariasamy
3
1
Centre for Clinical Epidemiology, Institute for Clinical Research, National Institutes of Health, Ministry of Health Malaysia, Shah Alam 40170, Selangor, Malaysia
2
Digital Health Research & Innovation, Institute for Clinical Research, National Institutes of Health, Ministry of Health Malaysia, Shah Alam 40170, Selangor, Malaysia
3
Institute for Clinical Research, National Institutes of Health, Ministry of Health Malaysia, Shah Alam 40170, Selangor, Malaysia
*
Author to whom correspondence should be addressed.
Publications 2022, 10(4), 36; https://doi.org/10.3390/publications10040036
Submission received: 26 April 2022 / Revised: 1 July 2022 / Accepted: 28 July 2022 / Published: 3 October 2022
Browse Figure
Versions Notes

Abstract

Background: The publication rates of abstracts after they were presented at the National Conference for Clinical Research (NCCR), a scientific conference held in Malaysia, was determined to gauge the scientific value of the conference, whilst providing comparative information with other scientific conferences. Methods: All the abstracts that were presented at the NCCR from 2014 to 2016 were analysed. Keywords from the abstract title, along with the first, second, and last author’s name, were searched via PubMed, Google Scholar, and Scopus to determine publication status. Results: A total of 320 abstracts were analysed. Of those, 57 abstracts (17.8%) were published. Almost 70% of published abstracts appeared in open access journals that charge article processing fees. Early publications (≤18 months from the conference date) had higher median journal impact factors compared to later publications. Approximately 42% of the published abstracts had collaborations with the Institute for Clinical Research (ICR) or Clinical Research Centres (CRCs). An increasing number of authors in an abstract and having the first author from a research centre, reduced and increased the odds of publication, respectively. Conclusions: The NCCR publication rate is lower compared to the reported average in other scientific conferences abroad. More encouragement and support to publish should be provided to the presenting authors. Clinicians should also be encouraged to collaborate with research centres such as those from the ICR or CRCs to boost publication likelihoods.

1. Introduction

Scientific abstracts are submitted to scientific conferences to facilitate dissemination of novel findings among members of a similar fraternity before being subjected to peer-reviewers for publication. However, abstracts may lack details pertaining to the methodology or present findings that are based on preliminary results [1,2]. The abstracts are also not subjected to a rigorous peer-review process that is commonly applied to full paper publications in scientific journals. These factors can prevent the translation of study findings into actual clinical practice. Additionally, scientific abstracts are often collated as conference proceedings, which may be difficult to locate and cite, thus reducing the outreach of the presented findings. Hence, presented abstracts should ideally be timely published to ensure the global sharing of these findings which may help shape subsequent scientific research or translation from research into practice. However, reports have shown that only a proportion of these presented abstracts achieve publication in a peer-reviewed journal.
A recent Cochrane systematic review that investigated 425 reports has shown average publication rates in scientific conferences reaching close to 40% [3]. Scientific conferences that are related to the medical or healthcare field often report publication rates which are close to or higher than this value. For instance, close to or over 50% of abstracts that are presented at conferences related to general internal medicine, orthopaedics, and cardiology achieved publication [4,5,6,7]. However, several reports also document publication rates that were substantially lower than this value, ranging from less than 4% [8] to less than 30% [9,10].
The number of publications that a country produces is used to rank and compare a country’s scientific output [11]. According to a national report that employed bibliometrics to measure the country’s research productivity, Malaysia produced the highest number of publications among the Association of Southeast Asian Nations (ASEAN) [12]. However, Malaysia ranked the third lowest in terms of citations per proceedings. These observations highlight the importance of publications in raising a country’s international standards in terms of knowledge and innovative productivity, which is suggestive of opinions that the proportion of abstracts that are converted to publications in a conference serves as an indicator to reflect a conference’s scientific value [13]. Moreover, the conversion rate of abstracts to full paper publications is related to the generation of citable literature, which helps to improve the quality of a conference via assessments using bibliographic citations [14,15].
Malaysia hosts multiple scientific conferences annually. This includes the National Conference for Clinical Research (NCCR), which has been organized by the Institute for Clinical Research (ICR) under the National Institutes of Health (NIH), Ministry of Health (MOH) Malaysia, since 2007. This scientific conference aims to share research findings, promote the conduct of clinical research in Malaysia, and disseminate information for continuous professional development [16]. The conference has a reputable standing as it is attended by over 500 delegates from the healthcare sector and hosts many distinguished local and international speakers. The NCCR accepts abstracts from all medical disciplines. Additionally, apart from clinical research, abstracts presenting other forms of research such as biomedical, health systems, and behavioural research, are also accepted. The conference welcomes abstracts that are submitted by both medical and allied health professionals such as doctors, dentists, pharmacists, nurses, dieticians, and psychologists.
To our knowledge, only one study has reported on the abstract publication rate of a scientific conference that was held in Malaysia. Chan et al. investigated the publication rate of a local paediatric conference [17]. However, this study was published in 2002, which meant that the results are dated. Moreover, the reported conference was focused on a specific medical discipline. There are presently no studies which have investigated the publication rates of abstracts that are presented at recent national scientific conferences held in Malaysia, especially those which accept multidisciplinary abstracts. Based on these knowledge gaps, we conducted a study with the primary objective of determining the publication rates of abstracts after they were presented at the NCCR, given that this conference is held in Malaysia at a national level, and presents multidisciplinary abstracts that are related to the medical or healthcare field. We determined publication rates based on three conferences that were held over three consecutive years. As a secondary objective, factors that are associated with abstracts obtaining publication were analysed to understand the key drivers for publication and identify areas that could be improved to bolster publication likelihoods.

2. Methods

We conducted a retrospective cross-sectional study of abstracts that were presented in the NCCR that was held from 2014 to 2016. All the poster abstracts that were presented in these three consecutive conferences were analysed. Data extraction was initiated on the 1st of October 2020 and completed on the 30th of October 2020. We only included publications that were published up to September 2020. This allowed a follow-up period of between four to six years from each conference date to allow sufficient time for publication, as noted in previous studies [18,19]. Given that we were investigating the publication rates post-conference, the presented abstracts that were published before the conference date were excluded based on a previous study [19].
We determined each abstract’s publication status by searching three databases, namely PubMed, Google Scholar, and Scopus; since it was recommended by a previous systematic review to utilize at least two databases when conducting a search [3]. Our search strategy involved using keywords from the abstract title along with the first, second, and last author’s name in a successive manner. Publication was ascertained if the title and content of the identified article is same or similar to the presented abstract, along with a matching author name. A successful publication was defined as a published full-length article in a peer-reviewed journal that was either indexed or non-indexed. After identifying abstracts achieving full paper publication, we further segregated these papers into publication periods that were either within 18 months, or above 18 months from the conference date. This cut-off period was selected because the longest average waiting time for a journal to publish a paper was almost 18 months [20]. This segregation provides a gauge into which abstracts were published in a timely manner after the conference. We traced the impact factor of the journal in which the article was published using the Journal Citation Reports database or from the journal website. We recorded the impact factor which corresponded to the year that the article was published.
Previous studies have noted that very good interobserver agreement exists when conducting a publication search [8,21]. Given that the total number of abstracts that was searched was considerably manageable, a single investigator (PXK) performed the entire search. When there was uncertainty over the publication status of the abstract due to certain content discrepancies, two investigators (NYLH and MAAR) were called to scrutinize and discuss the issue together with PXK before coming to a consensus. An odd number of investigators was selected for this task so that the third investigator can act as a tiebreaker should a dispute arise.
Apart from publication status, we categorised abstracts as being full research, case reports or case series, or audits or reports. Full research was defined as original research articles that were either systematic reviews or are empirical studies which employ proper quantitative or qualitative study designs. We also collected data on the study design, origin of institutions that were conducting the study, area of research, number and nature of collaborators, and the involvement of the ICR or Clinical Research Centres (CRCs) under the MOH. CRCs are research centres that serve as extended service branches of the ICR [22]. Currently there are 37 CRCs, each based in a public hospital located nationwide. CRCs serve to facilitate and regulate the conduct of clinical and any other health-related research that is held within the stationed hospital, and other health facilities within its zone of coverage. The ICR is also composed of several research centres that are based within the institute itself. To streamline the reporting of our findings, the involvement of ICR in presented abstracts was recorded as a CRC involvement throughout this study.
Randomized controlled trials (RCTs) have been shown to have higher likelihoods of publication due to its superior evidence generation capability [3]. Based on this observation, we further segregated the experimental study design of presented abstracts to allow publication rate comparisons between abstracts employing RCT as a study design, and all other abstracts. We also asked contactable primary authors for the reasons behind why RCT abstracts remained unpublished.

3. Statistical Analysis

Categorical variables were summarised using frequencies and percentages. Continuous variables were summarised using the median and interquartile range (IQR). Only the differences between the published and unpublished papers were tested using different statistical tests. Pearson’s chi-square test (cell counts ≥ 5) and Fisher’s exact test (cell counts < 5) were used to determine the associations between the categorical variables. The Wilcoxon rank sum test was used to compare the differences between the continuous variables. Univariate logistic regression was performed to assess the association of various factors with publications, whereby an odds ratio (OR) > 1 indicates increased odds of publication. In this study, p-values < 0.05 were considered significant. In view of a small sample size, variables in univariate regression with p-values less than 0.1 were selected for multivariate analysis. Statistical analysis was performed using R version 4.1.1.

4. Results

A total of 330 abstracts were presented over the three-year duration, from 2014 to 2016, at the NCCR. We excluded ten abstracts from analysis as they were published before the conference date in which the abstracts were presented. Of the 320 abstracts that were examined, only 57 abstracts (17.8%) were published as full-length manuscripts, out of which 63.2% obtained publication status within 18 months (Table 1). The number of presented abstracts increased from Year 2014 to Year 2015 and subsequently sloped in Year 2016. The publication rate was lowest in Year 2015 (11.6%) and highest in Year 2016 (27.8%). The average publication rate during these three consecutive years was 19.7% annually.
More than 75% (n = 251) of the presented abstracts were full research studies, with approximately 20% (n = 50) of these studies attaining publication. Case studies or case series accounted for the lowest number of abstracts, with less than 5% of the total abstracts that were presented. There was only one study protocol that was presented at the conference, and this was grouped together with the audit or report study category. In terms of study design, over 90% of the presented abstracts were observational studies. Experimental studies which consisted of RCTs and quasi-experiments, accounted for only 5% of the total abstracts. The remaining few abstracts were either systematic or scoping reviews, or other forms of research including study protocols, questionnaire development, and methodological research. Observational studies accounted for over 90% of the published articles. A total of 3.1% (n = 10) of presented abstracts were RCTs, with only two such abstracts attaining publication.
Most of the abstracts that were presented were based on research that was conducted by medical professionals, which consisted of doctors, dentists, and pharmacists. Research that was related to the general medical field accounted for almost 30% of the presented abstracts, and represented the highest proportion compared to other fields. Other fields that had substantial proportions of presented abstracts included surgical, healthcare practice and policy, women and children, and pharmacy. Around 34% of the presented abstracts involved researchers from CRCs, with slightly more than half of these abstracts having CRC researchers as a first author. Furthermore, over 40% of the published manuscripts had involvements with CRCs.
A large majority (81%, n = 260) of the presented abstracts were based on work that was conducted solely by staff from the Malaysian MOH. Conversely, less than 5% of the total presented abstracts had international collaborations, and these abstracts only accounted for 7% of the total published abstracts. More than 60% of the total published abstracts were accepted by open access peer-reviewed journals that require article processing charges. The median journal impact factor based on total published abstracts was 0.94 (IQR 0.45, 2.17).
Table 2 presents the timeline for all successfully published abstracts. More than 70% and 95% of these abstracts were published within two years and four years after the conference, respectively. The publication rate was highest during the first seven to 18 months after the conference, followed by a gradual decline thereafter. It took more than three years for six abstracts (10.5%) to reach publication. Univariate logistic regression in Table 3 shows that the number of authors was marginally associated with abstract publication. With every increase in authorship by one author, the odds of getting published dropped by 12%. This association was robust in the multivariate logistic regression model found in Table 4, whereby an increase in one author reduced the odds of publication by 14%. This model also showed that when an abstract’s first author was a CRC researcher, the odds of getting published was doubled.
Out of the eight unpublished abstracts employing RCT as a study design, the authors from seven of these abstracts responded to our inquiry and provided reasons for not publishing their research. A total of four of them cited reasons concerning lack of protected time or time constraints. Out of the four, two authors further cited that they were pre-occupied with further studies or subspeciality training, which further hampered efforts to write and placed publication as a lesser priority. The main study investigator leaving to another hospital was cited as an additional factor by another author. Similarly, one author explained that the main reason behind non-publication was because the main investigator left the public service to pursue a different career path. Another author’s study team decided not to publish because they felt that the research methodology that was used was questionable. Lastly, one author described nonproficiency in scientific writing as a major barrier towards achieving full paper publication.
Figure 1 displays the distribution of publications according to journal impact factor and time to publication for each conference year. The majority of the abstracts were published in journals with an impact factor of 0 to 2. There were two abstracts that were published in journals having an impact factor of more than 5; one was published within a year and the other took more than 30 months. The scatter plots for each year demonstrated that the impact factor was not correlated with the time taken to publish (Pearson’s product-moment correlation = −0.05, p-value = 0.714). However, Table 1 shows that the median impact factor of journals in which abstracts were published within 18 months after the conference was higher (1.04 (IQR 0.45, 2.17)) compared to abstracts that were published beyond that period (0.83 (IQR 0.40, 1.92)).

5. Discussion

Our study reports the publication rates of abstracts that were presented at a Malaysian national scientific conference which accepted multidisciplinary abstracts that spanned across clinical and other forms of research. Our findings serve as a comparator for other scientific conferences that are held within the region and abroad.
On average, about one fifth of all the abstracts that were presented at each NCCR were converted to full publication. This is approximately half of the average publication rate that was reported by the Cochrane systematic review [3]. This substantial contrast between the NCCR publication rates and most other scientific conferences may stem from several reasons. Firstly, the systematic review included international conferences that feature specific medical disciplines, which tend to yield high publication rates. For instance, scientific conferences for specific fields such as head and neck oncology, paediatric orthopaedics, orthopaedic surgery, hand surgery, and orthodontics recorded publication rates that ranged between 30% to 60% [23,24,25,26,27]. The NCCR accepts abstracts from various disciplines and does not focus on a particular medical field. Therefore, publishable abstracts from our local researchers could have been submitted to discipline-specific conferences to garner attention from an audience in a relevant fraternity to maximize the impact of research findings. Secondly, a survey among Malaysian clinicians found that only 16.9% had published a paper throughout their clinical practice [28]. As the NCCR is organized by a research institute under the Malaysian MOH, a significant proportion of the presented abstracts were submitted by Malaysian clinicians and other healthcare professionals. Hence, our low publication rates may reflect the low proportion of healthcare providers who actually publish. Lastly, only a few of the presented abstracts had international collaborations, which may have also contributed to lower publication rates. Publication productivity of Malaysian researchers in academia have been shown to be associated with conducting research that possess international collaborations [29]. This practice has also led to higher citation rates and publications in journals having higher impact factors [30]. Thus, our findings suggest the importance of encouraging local researchers to collaborate with international partners.
In spite of this contrast in publication rates, we observed that the distribution of abstracts that were published from the date of presentation at the NCCR over time was similar across many other medical and healthcare conferences [8,9,25,31]. This observation is most likely attributed to journals of such disciplines having similar timelines for the entire peer-review to publication process [32].
RCTs serve as a gold standard to study causal relationships between interventions and outcomes to derive a true estimate of effectiveness [33]. This characteristic makes it a superior study design for evaluating treatments in clinical research [34]. Hence, RCT abstracts may serve as a possible benchmark to steer scientific productivity amongst the clinical research fraternity. However, despite the NCCR being centrally themed as a clinical research focused conference, we found less than 4% of the presented abstracts applying RCT as a study design. This paucity of RCT studies is similarly observed in other conferences [31,35]. Such low numbers reflect the multitude of barriers that are faced by researchers when conducting RCT research, such as inadequate knowledge, funding issues, complex regulations, and infrastructure shortages [36]. Given the immense challenges and other disadvantages that are associated with conducting RCTs, researchers have suggested using other study designs which can adequately answer research questions without the need for an RCT [34,37]. Nevertheless, our findings highlight the need for more supportive action by the Malaysian MOH and other research-related agencies in driving successful RCT studies, which is in line with the NCCR’s main objective of promoting clinical research. Contrary to common observations that RCT abstracts have higher likelihoods of being published [3], only two out of the ten submitted RCT abstracts in our study were published. The reasons for non-publication were mainly due to time constraints and lesser prioritisations given to publish, which are commonly reported reasons among researchers who are working as healthcare professionals in the field [5,28,38]. Other reasons contributing to unpublished trials include the observation of undesirable results and protecting intellectual property rights [39,40]. Non-publication not only raises ethical issues arising from the futility of exposing trial subjects to potential risks and harms, but it also leads to bias in the true effectiveness of an intervention [41].
Abstract publication likelihood significantly improved when the lead author is affiliated to the ICR or a CRC. There are several possible factors for such observations. First authors are usually the main drivers to produce manuscripts for publication [42]. Hence, the motivation, determination, and focus to complete a full manuscript critically lies with the lead author. These key values differ between researchers and healthcare providers. Researchers are more motivated to publish articles to fulfil key performance requirements and for career advancement. They also tend to have more time to focus on completing a manuscript as it is part of their core duty. Furthermore, they may have more opportunities to receive proper training in scientific writing, which could boost the efficiency of producing a manuscript. Conversely, healthcare providers face multiple barriers when attempting to complete a full manuscript. As echoed by respondents who did not publish their RCT abstracts, clinical practice is often plagued with time-constraints and centres upon patient care as a core duty, leading to insufficient time to focus on scientific writing [28]. Moreover, as cited by two of our respondents, the migration or posting of personnel to other facilities could disrupt the ability to focus on publishing a research work. Therefore, steps should be taken to structurally reengineer the work system to improve scientific productivity among healthcare providers, which could include the allocation of sufficient protected time for writing and more opportunities for training in scientific writing.
Full research abstracts are usually associated with higher chances of publication as compared to case reports, case series, audits, and reports. Full research abstracts often feature articles that employ study designs to answer specific research questions using sufficient sample sizes. These characteristics produce empirical evidence that is supported by statistical or systematic qualitative analyses, thus increasing the chances of obtaining publication as compared to other study types [3]. Case reports, case series, audits, and reports that have little or no research elements, face additional challenges that are associated with publishing, which may lead to higher likelihoods of rejection [43,44]. However, this association was not significant in our study. This may be due to the generally lower rates of publications that are observed among all presented abstracts, irrespective of study category. The paltry conversion rate that was observed among RCT abstracts in the conference echoes this point. These findings further highlight again the time constraints and lesser prioritisations to publish as possible systemic issues stifling efforts to publish, since conference attendees are mostly from the MOH.
Current research trends favour increasing the number of authors to address increasingly complex research questions that require contributions from various fields or to confer honorary authorships to allow for a project’s conduct [45]. However, our findings show that an increasing number of authors seemed to be detrimental to the publication likelihood of a submitted abstract. Although we did not evaluate the possible reasons for non-publication, we postulate that the number of authors reflect the complexity of the research. Complex research may face many challenges such as implementing scientifically valid methods based on current available resources, applying appropriate study designs, and collecting sufficient sample sizes. These challenges have been cited as some of the reasons for unpublished abstracts among Singaporean anaesthesiologists [46].
Our study found that abstracts that were published beyond 18 months from the conference date were published in journals with a lower impact factor. This observation may stem from the time pressure that authors face when attempting to publish study findings in a timely manner. Study findings are best disseminated as early as possible in order to preserve the validity and relevance of the information that is generated from the study results. However, a submitted manuscript for publication may encounter several forms of delays, including experiencing a waiting period during the peer-review process, time taken for manuscript revisions, and outright rejections [32,47]. This is further aggravated when authors practice journal shopping in an attempt to publish in journals with a high impact factor [48]. Publication delays may also arise from manuscripts which are slow to be completed. In order to maintain the relevancy of research findings while compensating for publication delays, authors may have been compelled to select journals with a lower impact factor as time passes after the conference to ease and speed up the publication process [48,49].
The low number of research abstracts and publications by the allied health fraternity highlights the possible barriers that are faced by this group of researchers. Apart from having to prioritize clinical roles, studies have shown that allied health professionals face formidable hurdles to conduct research due to insufficient research skills, time, funding, and support from managers [50,51]. These challenges might pose a threat towards advancing clinical research, given that allied health personnel spend longer periods of time with patients compared to other healthcare professionals, and thus may be more capable to identify research gaps that potentially answer real-life problems. Hence, strategic capacity building measures that are guided by an integrated framework is needed to encourage research participation from allied health professionals in order to leverage their potential to conduct impactful research [52]. Such strategies include organising research training programmes, coaching from experienced researchers, and providing incentives to conduct and publish research.

6. Limitations

Our study has several limitations. Although publication status was determined by searching three databases, there was still a possibility that published abstracts were not found in these databases. Moreover, the observation period between the conference date and data collection period was limited to only four to six years. Apart from authors who conducted RCTs, none of the authors were contacted to verify publication status and to determine the underlying reasons behind non-publication. Therefore, the publication rates may be underestimated. Factors that led to the various publication timelines were also not explored.
Data analysis only involved the use of submitted abstracts to each conference. Due to abstracts having insufficient detailed information regarding study conduct and results, we were unable to conduct objective quality assessments for each abstract to determine the extent of quality predicting publication. Due to resource constraints, we also did not analyse how study characteristics such as sample size, nature of study findings, and multicentric studies, affected publication likelihoods. Due to similar reasons, we did not analyse the differences between the data that were presented in abstracts compared to the published version.

7. Further Work

This study has provided a gauge on the scientific value of the NCCR based on the publication rates of presented abstracts and highlighted certain areas for improvement. Nevertheless, there are other factors which have been shown to contribute towards the quality of a scientific conference. For instance, subjective assessments via consultations with specialists in the field, or the use of citation analysis using existing journal metrics, have been used to determine conference quality [14,15]. Other factors contributing to quality include the use of factors such as longevity, popularity, prestige, conference size, periodicity, and data coverage [15]. These indicators should be explored among local conferences in Malaysia and the wider region so that the quality of held conferences can be thoroughly determined to spur competitiveness and advancement. Periodic assessments of the NCCR publication rates for recent and future conferences should also be conducted to assess the developmental progress of the conference.
The reasons that were disclosed by authors from unpublished RCT abstracts in our study offered a glimpse into the potential barriers towards publication efforts. Future research should further explore the reasons for other forms of unpublished research so that amenable factors can be acted upon to boost the conference’s scientific productivity. Mechanisms that can afford clinicians protected time to be actively involved in publication efforts without jeopardising healthcare services should also be explored. Lastly, it will be valuable to determine if the submitted abstracts and research publications adhere to available reporting guidelines such as PRISMA, CONSORT, or STROBE, and whether this factor predicts for publication or journal impact factor.

8. Conclusions

We observed a publication rate of 17.8% based on the total number of abstracts that were presented over three consecutive years at the NCCR. Given that this rate is lower compared to the reported average in other scientific conferences abroad, more encouragement and support to publish should be provided to authors who submit abstracts to the NCCR. These may include assistance in the form of training, funding, and incentives that promote publication. There needs to be additional attention paid towards promoting and supporting the conduct of RCTs. Efforts should also be made to encourage the publication of RCTs and allied health research. Lastly, the benefits of fostering collaborations and receiving support from research centres such as those from the ICR or CRCs to enhance publication likelihood should be promoted, especially among healthcare providers who may be too preoccupied with clinical roles to complete manuscripts for publication.

Author Contributions

K.M.P. conceived the project. N.Y.L.H., M.D.P. and M.A.A.R. wrote the study protocol. N.Y.L.H., P.X.K. and M.A.A.R. collected the data. X.C.W. analysed the data and generated visualizations. N.Y.L.H., P.X.K. and X.C.W. interpreted the results. N.Y.L.H. wrote the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

No funding was received for conducting this study.

Data Availability Statement

The dataset that was used for this study belongs to the Ministry of Health, Malaysia. Hence, the dataset may be available from the corresponding author via a formal request through relevant authorities at the Ministry of Health, Malaysia.

Acknowledgments

We would like to thank the Director General of Health Malaysia for his permission to publish this article. We would also like to thank Wei Hong Lai and Siew Cheng Looi for providing records of abstracts that were presented during past NCCR events.

Conflicts of Interest

Non-financial interests: All authors are affiliated with the Institute for Clinical Research and have been involved in organizing past and present NCCR events.

Abbreviations

ASEANAssociation of Southeast Asian Nations
CRCClinical Research Centre
ICRInstitute for Clinical Research
ICMJEInternational Committee of Medical Journal Editors
MOHMinistry of Health
NCCRNational Conference for Clinical Research
RCTRandomised controlled trial

References

  1. Hopewell, S.; Clarke, M. Abstracts Presented at the American Society of Clinical Oncology Conference: How Completely Are Trials Reported? Clin. Trials 2005, 2, 265–268. [Google Scholar] [CrossRef] [PubMed]
  2. Hopewell, S.; Clarke, M.; Askie, L. Reporting of Trials Presented in Conference Abstracts Needs to Be Improved. J. Clin. Epidemiol. 2006, 59, 681–684. [Google Scholar] [CrossRef] [PubMed]
  3. Scherer, R.; Meerpohl, J.; Pfeifer, N.; Schmucker, C.; Schwarzer, G.; Von Elm, E. Full Publication of Results Initially Presented in Abstracts. Cochrane Database Syst. Rev. 2018, 11, MR000005. [Google Scholar] [CrossRef] [PubMed]
  4. Egloff, H.M.; West, C.P.; Wang, A.T.; Lowe, K.M.; Varayil, J.E.; Beckman, T.J.; Sawatsky, A.P. Publication Rates of Abstracts Presented at the Society of General Internal Medicine Annual Meeting. J. Gen. Intern. Med. 2017, 32, 673–678. [Google Scholar] [CrossRef]
  5. Balhatchet, B.; Schütze, H.; Awais, A.; Williams, N. Publication Rate of Abstracts Presented at the Australian Orthopaedic Association Annual Scientific Meeting. ANZ J. Surg. 2022, 92, 493–498. [Google Scholar] [CrossRef]
  6. Bovonratwet, P.; Webb, M.L.; Ondeck, N.T.; Shultz, B.N.; McLynn, R.P.; Cui, J.J.; Grauer, J.N. High Publication Rate of Abstracts Presented at Lumbar Spine Research Society Meetings. Int. J. Spine Surg. 2018, 12, 713–717. [Google Scholar] [CrossRef]
  7. Fosbøl, E.L.; Fosbøl, P.L.; Harrington, R.A.; Eapen, Z.J.; Peterson, E.D. Conversion of Cardiovascular Conference Abstracts to Publications. Circulation 2012, 126, 2819–2825. [Google Scholar] [CrossRef]
  8. Komagamine, J.; Yabuki, T. Full-Text Publication Rate of Abstracts Presented at the Japan Primary Care Association Annual Meetings (2010–2012): A Retrospective Observational Study. BMJ Open 2018, 8, 4–9. [Google Scholar] [CrossRef]
  9. Akkoc, I. Publication Rates of Abstracts Presented at the World Congress on Pain Held by the International Association for the Study of Pain in 2010. J. Int. Med. Res. 2019, 31, 1–9. [Google Scholar] [CrossRef]
  10. Khajehnoori, M.; Stupart, D.; Watters, D. Publication Rate of General Surgery Abstracts Presented at the Royal Australasian College of Surgeons Annual Scientific Congress. ANZ J. Surg. 2018, 88, 16–19. [Google Scholar] [CrossRef]
  11. Scimago Journal & Country Rank. Available online: https://www.scimagojr.com/countryrank.php (accessed on 29 June 2022).
  12. Ministry of Energy, Science, Technology, Environment and Climate Change. National Bibliometric Study (2001–2017); Malaysian Science and Technology Information Centre: Putrajaya, Malaysia, 2018; pp. 1–33.
  13. Daruwalla, Z.J.; Huq, S.S.; Wong, K.L.; Nee, P.Y.; Murphy, D.P. “Publish or Perish”—Presentations at Annual National Orthopaedic Meetings and Their Correlation with Subsequent Publication. J. Orthop. Surg. Res. 2015, 10, 58. [Google Scholar] [CrossRef] [PubMed]
  14. Meho, L.I. Using Scopus’s CiteScore for Assessing the Quality of Computer Science Conferences. J. Informetr. 2019, 13, 419–433. [Google Scholar] [CrossRef]
  15. Martins, W.S.; Gonçalves, M.A.; Laender, A.H.F.; Ziviani, N. Assessing the Quality of Scientific Conferences Based on Bibliographic Citations. Scientometrics 2010, 83, 133–155. [Google Scholar] [CrossRef]
  16. Chan, S.T.; Hor, C.P.; Beh, K.M.K.; Mak, W.Y. CRC Newsletter; Goh, P.P., Ch’ng, C.C., Eds.; Clinical Research Centre; 2014; pp. 10–11. Available online: https://www.crc.gov.my/wp-content/uploads/documents/newsletter/03_crc_newsletter3_2014.pdf (accessed on 20 April 2022).
  17. Chan, P.W.K.; Norzila, M.Z.; Bilkis, A.A.; Mazidad, A. Do Paediatric Research Presentations at Local Meetings Get Published? Med. J. Malays. 2002, 57, 482–486. [Google Scholar]
  18. Song, F.; Parekh, S.; Hooper, L.; Loke, Y.K.; Ryder, J.; Sutton, A.J.; Hing, C.; Kwok, C.S.; Pang, C.; Harvey, I. Dissemination and Publication of Research Findings: An Updated Review of Related Biases. Health Technol. Assess. 2010, 14, 1–193. [Google Scholar] [CrossRef]
  19. Waldorff, F.B.; Petersen, K.; Vinther, S.; Sandholdt, H.; Siersma, V.; Andersen, J.S. Full Journal Publication of Abstracts Presented at the Nordic Congress of General Practice in 2009 and 2011. Scand. J. Prim. Health Care 2017, 35, 84–88. [Google Scholar] [CrossRef]
  20. Björk, B.-C.; Solomon, D. The Publishing Delay in Scholarly Peer-Reviewed Journals. J. Informetr. 2013, 7, 914–923. [Google Scholar] [CrossRef]
  21. Murrey, D.B.; Wright, R.W.; Seiler, J.G.; Day, T.E.; Schwartz, H.S. Publication Rates of Abstracts Presented at the 1993 Annual Academy Meeting. Clin. Orthop. Relat. Res. 1999, 359, 247–253. [Google Scholar] [CrossRef]
  22. Institute for Clinical Research Vision, Mission & Function. Available online: https://www.crc.gov.my/vision-mission-function/ (accessed on 11 April 2022).
  23. North, A.S.; Ur-Rehman, J.; Lin, D.J. Trends and Publication Rates of Abstracts Presented at the British Association of Head and Neck Oncologists’ (BAHNO) Annual Meetings: 2009–2015. Br. J. Oral Maxillofac. Surg. 2020, 58, e296–e300. [Google Scholar] [CrossRef]
  24. Rawicki, N.L.; Fasani-Feldberg, G.B.; Agarwalla, A.; Sethuraman, S.A.; Pammal, R.S.; Lapow, J.; Cristofaro, R. Publication Rates of Abstracts Presented at Pediatric Orthopaedic Society of North America Meetings Between 2013 and 2016. J. Pediatr. Orthop. 2022, 42, 59–63. [Google Scholar] [CrossRef]
  25. Donegan, D.J.; Kim, T.W.; Lee, G.-C. Publication Rates of Presentations at an Annual Meeting of the American Academy of Orthopaedic Surgeons. Clin. Orthop. Relat. Res. 2010, 468, 1428–1435. [Google Scholar] [CrossRef] [PubMed]
  26. Peake, M.; Rotatori, R.M.; Ovalle, F.; Gobble, R.M. Publishing Conversion Rates and Trends in Abstracts Presented at the American Association for Hand Surgery Annual Meeting: A 5 Years Review. HAND 2019, 16, 397–401. [Google Scholar] [CrossRef] [PubMed]
  27. Al-Yatama, F.; Pandis, N.; Cobourne, M.T.; Seehra, J. Publication Rate of Abstracts from Presentations at the British Orthodontic Conference 2009–2014. J. Orthod. 2020, 47, 311–319. [Google Scholar] [CrossRef] [PubMed]
  28. Ganasegeran, K.; Ch’ng, A.S.H.; Amar Jamil, M.F.; Looi, I. Clinicians’ Publication Output : Self-Report Survey and Bibliometric Analysis. Publications 2020, 8, 15. [Google Scholar] [CrossRef]
  29. Bakri, A.; Azura, N.M.; Nadzar, M.; Ibrahim, R.; Tahira, M. Publication Productivity Pattern of Malaysian Researchers in Scopus from 1995 to 2015. J. Scientometr. Res. 2017, 6, 86–101. [Google Scholar] [CrossRef]
  30. Low, W.Y.; Ng, K.H.; Kabir, M.A.; Koh, A.P.; Sinnasamy, J. Trend and Impact of International Collaboration in Clinical Medicine Papers Published in Malaysia. Scientometrics 2014, 98, 1521–1533. [Google Scholar] [CrossRef]
  31. Komagamine, J.; Kobayashi, M. Publication Rate of Abstracts Presented at Japan Geriatrics Society Annual Meetings (2011–2012): A Retrospective Observational Study. BMC Res. Notes 2018, 11, 36. [Google Scholar] [CrossRef]
  32. Huisman, J.; Smits, J. Duration and Quality of the Peer Review Process: The Author’s Perspective. Scientometrics 2017, 113, 633–650. [Google Scholar] [CrossRef]
  33. Hariton, E.; Locascio, J.J. Randomised Controlled Trials—The Gold Standard for Effectiveness Research: Study Design: Randomised Controlled Trials. BJOG An Int. J. Obstet. Gynaecol. 2018, 125, 1716. [Google Scholar] [CrossRef]
  34. Nabi, G. Randomised Controlled Trials in Medical Research: Do We Need Alternatives? Scott. Med. J. 2020, 65, 1–2. [Google Scholar] [CrossRef]
  35. Trost, M.; Langer, F.; Lechler, P.; Schröder, F.; Wetterkamp, M.; Schulte, T.L.; Eysel, P.; Boese, C.K. Publication Rate of Abstracts Presented at the Congress of the European Federation of National Associations of Orthopaedics and Traumatology (EFORT). Orthop. Traumatol. Surg. Res. 2019, 105, 1453–1457. [Google Scholar] [CrossRef] [PubMed]
  36. Djurisic, S.; Rath, A.; Gaber, S.; Garattini, S.; Bertele, V.; Ngwabyt, S.N.; Hivert, V.; Neugebauer, E.A.M.; Laville, M.; Hiesmayr, M.; et al. Barriers to the Conduct of Randomised Clinical Trials within All Disease Areas. Trials 2017, 18, 360. [Google Scholar] [CrossRef] [PubMed]
  37. Pearce, W.; Raman, S.; Turner, A. Randomised Trials in Context: Practical Problems and Social Aspects of Evidence-Based Medicine and Policy. Trials 2015, 16, 394. [Google Scholar] [CrossRef] [PubMed][Green Version]
  38. Abrams, M.; Bruijns, S.R.; van Hoving, D.J. Dissemination Patterns of Scientific Abstracts Presented at the First and Second African Conference of Emergency Medicine. Afr. J. Emerg. Med. 2019, 9, 101–105. [Google Scholar] [CrossRef] [PubMed]
  39. Blumenthal, D.; Campbell, E.G.; Anderson, M.S.; Causino, N.; Louis, K.S. Withholding Research Results in Academic Life Science. Evidence from a National Survey of Faculty. JAMA 1997, 277, 1224–1228. [Google Scholar] [CrossRef]
  40. Campbell, E.G.; Clarridge, B.R.; Gokhale, M.; Birenbaum, L.; Hilgartner, S.; Holtzman, N.A.; Blumenthal, D. Data Withholding in Academic Genetics: Evidence from a National Survey. JAMA 2002, 287, 473–480. [Google Scholar] [CrossRef]
  41. Wallach, J.D.; Krumholz, H.M. Not Reporting Results of a Clinical Trial Is Academic Misconduct. Ann. Intern. Med. 2019, 341, 293–294. [Google Scholar] [CrossRef]
  42. Bhattacharya, S. Authorship Issue Explained. Indian J. Plast. Surg. 2010, 43, 233–234. [Google Scholar] [CrossRef]
  43. Ul Haq, R.; Dhammi, I.K. Effective Medical Writing: How to Write a Case Report Which Editors Would Publish. Indian J. Orthop. 2017, 51, 237–239. [Google Scholar] [CrossRef]
  44. Alderson, D. What to Publish. Ann. R. Coll. Surg. Engl. 2015, 97, 169–172. [Google Scholar] [CrossRef]
  45. Aboukhalil, R. The Rising Trend in Authorship; The Winnower: London, UK, 2014; p. e141832. [Google Scholar] [CrossRef]
  46. Thu Truong, H.T.; Goy, R.W.L.; Sultana, R.; Koh, D.L.K.; Singh, P.A.; Siow, Y.N.; Soh, C.R.; Sng, B.L. Publication Outcomes after Conference Abstract Submissions in a Singapore Anesthesiology Academic Clinical Program. Bali J. Anesthesiol. 2021, 5, 102–107. [Google Scholar] [CrossRef]
  47. Luft, H.S. From the Editors: Perspectives on Turnaround Time. Health Serv. Res. 2004, 39, 1–6. [Google Scholar] [CrossRef] [PubMed]
  48. Powell, K. Does It Take Too Long to Publish Research? Nature 2016, 530, 148–151. [Google Scholar] [CrossRef] [PubMed]
  49. Paulus, F.M.; Cruz, N.; Krach, S. The Impact Factor Fallacy. Front. Psychol. 2018, 9, 1487. [Google Scholar] [CrossRef] [PubMed]
  50. Pager, S.; Holden, L.; Golenko, X. Motivators, Enablers, and Barriers to Building Allied Health Research Capacity. J. Multidiscip. Healthc. 2012, 5, 53–59. [Google Scholar] [CrossRef]
  51. Borkowski, D.; McKinstry, C.; Cotchett, M.; Williams, C.; Haines, T. Research Culture in Allied Health: A Systematic Review. Aust. J. Prim. Health 2016, 22, 294–303. [Google Scholar] [CrossRef]
  52. Matus, J.; Walker, A.; Mickan, S. Research Capacity Building Frameworks for Allied Health Professionals—A Systematic Review. BMC Health Serv. Res. 2018, 18, 716. [Google Scholar] [CrossRef]
Publications 10 00036 g001
Figure 1. Scatter plot displaying the relationship of journal impact factor against the duration from conference to publication (Months) for the published abstracts that were presented at the 2014, 2015, and 2016 NCCR.
Figure 1. Scatter plot displaying the relationship of journal impact factor against the duration from conference to publication (Months) for the published abstracts that were presented at the 2014, 2015, and 2016 NCCR.
Publications 10 00036 g001
Table 1. Factors associated with publication among abstracts that were presented at the 2014 to 2016 NCCR.
Table 1. Factors associated with publication among abstracts that were presented at the 2014 to 2016 NCCR.
VariableOverall, n= 320 1No, n = 263 1Yes, n = 57 1p-Value 2p-Value 3
Overall, n= 57 1≤18 months,
n = 36 1		>18 months,
n = 21 1
NCCR Year 0.0080.63
201486 (26.8%)69 (26.2%)17 (29.8%)10 (27.8%)7 (33.3%)
2015155 (48.4%)137 (52.1%)18 (31.6%)13 (36.1%)5 (23.8%)
201679 (24.6%)57 (21.7%)22 (38.6%)13 (36.1%)9 (42.9%)
Study categories 0.170.83
Audit/Report/Protocol54 (16.9%)49 (18.6%)5 (8.8%)4 (11.1%)1 (4.8%)
Case Study or Case Series15 (4.7%)13 (4.9%)2 (3.5%)1 (2.8%)1 (4.8%)
Full Research251 (78.4%)201 (76.4%)50 (87.7%)31 (86.1%)19 (90.5%)
Research types 0.69>0.99
RCT 10 (3.1%)8 (3.0%)2 (3.5%)1 (2.8%)1 (4.8%)
Others 4310 (96.9%)255 (97.0%)55 (96.5%)35 (97.2%)20 (95.2%)
Study design 0.880.41
Experimental16 (5.0%)13 (4.9%)3 (5.3%)1 (2.8%)2 (9.5%)
Observational297 (92.8%)245 (93.2%)52 (91.2%)34 (94.4%)18 (85.7%)
Systematic/scoping review4 (1.3%)3 (1.1%)1 (1.8%)0 (0%)1 (4.8%)
Others3 (0.9%)2 (0.8%)1 (1.8%)1 (2.8%)0 (0%)
Research fields 5
Diagnostic medicine13 (4.1%)11 (4.2%)2 (3.5%)1 (2.8%)1 (4.8%)0.500.43
Emergency & critical care14 (4.4%)12 (4.6%)2 (3.5%)2 (5.6%)0 (0%)
Healthcare practice & policy47 (14.7%)38 (14.4%)9 (15.8%)5 (13.9%)4 (19.0%)
Medical95 (29.7%)79 (30.0%)16 (28.1%)11 (30.6%)5 (23.8%)
Pharmacy41 (12.8%)34 (12.9%)7 (12.3%)2 (5.6%)5 (23.8%)
Surgical48 (15.0%)36 (13.7%)12 (21.1%)8 (22.2%)4 (19.0%)
Women & children45 (14.1%)36 (13.7%)9 (15.8%)7 (19.4%)2 (9.5%)
Others17 (5.3%)17 (6.5%)0 (0%)--
Profession 0.220.075
Medical Professionals272 (85.0%)221 (84.0%)51 (89.5%)30 (83.3%)21 (100%)
Allied Health14 (4.4%)14 (5.3%)0 (0%)--
Combination34 (10.6%)28 (10.6%)6 (10.5%)6 (16.7%)0 (0%)
Number of collaborators2 (1, 3)2 (1, 3)2 (1, 3)2 (1, 2)2 (1, 3)0.650.064
Involving researchers from CRC110 (34.4%)86 (32.7%)24 (42.1%)17 (47.2%)7 (33.3%)0.180.31
First author from CRC61 (19.0%)45 (17.1%)16 (28.1%)13 (36.1%)3 (14.3%)0.0560.077
Number of authors4 (3, 6)4 (3, 6)4 (3, 5)4 (2, 5)4 (3, 5)0.170.42
Number of authors (≤4)179 (55.9%)145 (55.1%)34 (59.6%)23 (63.9%)11 (52.4%)0.530.39
Institutes involved 0.420.43
MOH only260 (81.3%)217 (82.5%)43 (75.4%)29 (80.6%)14 (66.7%)
MOE only13 (4.1%)10 (3.8%)3 (5.3%)2 (5.6%)1 (4.8%)
Others47 (14.7%)36 (13.7%)11(19.3%)5 (13.9%)6 (28.6%)
International collaborators15 (4.7%)11 (4.2%)4 (7.0%)1 (2.8%)3 (14.3%)0.320.14
Open access journal charging fees 36 (63.2%)25(69.4%)11 (52.4%) 0.20
Journal impact factor 0.94 (0.45, 2.17)1.04 (0.45, 2.17)0.83 (0.40, 1.92) 0.72
Missing data 6 633
Time to publication (Days) 444 (274, 814)320 (216, 414)910 (793, 1399) -
Time to publication (Months) 14 (9, 26)10 (7, 13)29 (26, 45) -
1 Median (IQR) or Frequency (%), 2 Pearson’s Chi-square test; Fisher’s exact test; Wilcoxon rank sum test; p-value only for published and not published, 3 Pearson’s Chi-square test; Fisher’s exact test; Wilcoxon rank sum test; p-value only for published >18 months or ≤ 18 months, 4 These include observational studies, quasi-experimental studies, systematic and scoping reviews, and other study designs, 5 Due to computational power, Fisher’s exact test was done with simulated p-value, 6 Publications in non-indexed journals were categorised as missing data when analysing journal impact factor of published abstracts because non-indexed journals do not have an impact factor.
Table 2. Distribution of 57 published abstracts by time from presentation at the 2014 to 2016 NCCR.
Table 2. Distribution of 57 published abstracts by time from presentation at the 2014 to 2016 NCCR.
Time from the ConferenceNumber of PublicationsAccumulated Number of Publications, n (%)
Published before NCCR10-
0–6 months8 (14.0%)8 (14.0%)
7–12 months16 (28.1%)24 (42.1)
13–18 months12 (21.1%)36 (63.2)
19–24 months4 (7.0%)40 (70.2)
25–30 months7 (12.3%)47 (82.5)
31–36 months4 (7.0%)51 (89.5)
37–42 months0 (0.0%)51 (89.5)
43–48 months4 (7.0%)55 (96.5)
49 months or later2 (3.5%)57(100)
Table 3. Factors associated with publication based on univariate logistic regression model.
Table 3. Factors associated with publication based on univariate logistic regression model.
CharacteristicOR 195% CI 1p-Value
Number of authors0.880.76, 1.000.046
Study categories 0.11
Audit/Report/Protocol——
Case Study or Case Series1.310.17, 6.78
Full Research2.461.02, 7.37
Research types 0.86
Others 2——
RCT1.160.17, 4.78
Study design 0.90
Observational--
Experimental1.090.30, 3.95
Systematic/scoping review1.570.16, 15.40
Others2.360.21, 26.47
Research fields 0.28
Diagnostic Medicine——
Emergency & Critical care0.920.10, 8.74
Healthcare practice & policy1.300.28, 9.37
Medical1.110.26, 7.64
Pharmacy1.130.23, 8.35
Surgical1.830.41, 12.9
Women & Children1.380.30, 9.90
Profession 0.28
Others——
Medical professionals alone1.620.70,4.41
Involving researchers from CRC 0.18
No——
Yes1.500.83, 2.68
First author from CRC 0.066
No——
Yes1.890.96, 3.62
Institution involved 0.38
MOE only——
MOH only0.660.19, 3.03
Others1.020.26, 5.13
1 OR = Odds Ratio, CI = Confidence Interval, 2 These include observational studies, quasi-experimental studies, systematic and scoping reviews, and other study designs.
Table 4. Factors associated with publication based on multivariate logistic regression model.
Table 4. Factors associated with publication based on multivariate logistic regression model.
CharacteristicOR 195% CI 1p-Value
Number of authors0.860.75, 0.980.032
First author from CRC 0.029
No——
Yes2.131.06, 4.15
1 OR = Odds Ratio, CI = Confidence Interval.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Hing, N.Y.L.; Wong, X.C.; Kuan, P.X.; Pathmanathan, M.D.; Abdul Rahman, M.A.; Peariasamy, K.M. Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference. Publications 2022, 10, 36. https://doi.org/10.3390/publications10040036

AMA Style

Hing NYL, Wong XC, Kuan PX, Pathmanathan MD, Abdul Rahman MA, Peariasamy KM. Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference. Publications. 2022; 10(4):36. https://doi.org/10.3390/publications10040036

Chicago/Turabian Style

Hing, Nicholas Yee Liang, Xin Ci Wong, Pei Xuan Kuan, Mohan Dass Pathmanathan, Mohd Aizuddin Abdul Rahman, and Kalaiarasu M. Peariasamy. 2022. "Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference" Publications 10, no. 4: 36. https://doi.org/10.3390/publications10040036

APA Style

Hing, N. Y. L., Wong, X. C., Kuan, P. X., Pathmanathan, M. D., Abdul Rahman, M. A., & Peariasamy, K. M. (2022). Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference. Publications, 10(4), 36. https://doi.org/10.3390/publications10040036

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop