Next Article in Journal
Quick Analysis of Organic Amendments via Portable X-ray Fluorescence Spectrometry
Next Article in Special Issue
The Willingness for Downward Referral and Its Influencing Factors: A Cross-Sectional Study among Older Adults in Shandong, China
Previous Article in Journal
Factors Associated with Sleep Disorders among Methadone-Maintained Drug Users in Vietnam
Previous Article in Special Issue
Effect of an Integrated Payment System on the Direct Economic Burden and Readmission of Rural Cerebral Infarction Inpatients: Evidence from Anhui, China
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 16
Issue 22
10.3390/ijerph16224316
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Effect of the Full Coverage of Essential Medicines Policy on Utilization and Accessibility of Primary Healthcare Service for Rural Seniors: A Time Series Study in Qidong, China

by
Ying Wang
1,
Yulei Zhu
1,
Hang Shi
1,
Xiaoluan Sun
1,
Na Chen
2 and
Xin Li
1,3,4,*
1
Department of Health Policy, School of Health Policy and Management, Nanjing Medical University, Nanjing 211166, China
2
School of Health Economics and Management, Nanjing University of Chinese Medicine, Nanjing 211166, China
3
Department of Clinical Pharmacy, School of Pharmacy, Nanjing Medical University, Nanjing 211166, China
4
Center for Global Health, School of Public Health, Nanjing Medical University, Nanjing 211166, China
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2019, 16(22), 4316; https://doi.org/10.3390/ijerph16224316
Submission received: 20 September 2019 / Revised: 27 October 2019 / Accepted: 3 November 2019 / Published: 6 November 2019
(This article belongs to the Special Issue Recent Advances in Healthcare Services in China)
Browse Figures
Review Reports Versions Notes

Abstract

:
Background: Since 2015, in order to handle the increasing prevalence of age-related diseases and escalating health expenditures arising from the aging population, the full coverage of essential medicines (FCEMs) policy for rural seniors has been implemented in primary healthcare institutions of Qidong County of Jiangsu, China. The purpose of this study is to examine the long-term effects of the introduction of FCEMs’ policy on the utilization and accessibility of primary healthcare service for elderly beneficiaries. Methods: The retrospective study was conducted in Qidong County in the Jiangsu province, China. A 47-month longitudinal dataset involving 91,444 health insurance claims records of inpatients aged 70 and older in primary healthcare institutions was analyzed. Changes in health service utilization (average length of stay), patient copayments (out-of-pocket expenses), New Rural Cooperative Medical System (NRCMS) reimbursement rate and daily hospitalization costs per patient were analyzed using interrupted time series analysis. Augment Dicky-Fuller unit root method was used to test the stationarity of the series alongside the Durbin Watson method to test autocorrelation. Results: Average length of stay increased at 0.372 bed-days per month before the implementation of FCEMs policy, whereas the increasing trend was slowed down at 0.003 bed-days per month after the implementation of FCEMs policy (p < 0.001). The average out-of-pocket expenses increased by 38.035 RMB monthly in pre-implementation of the policy period, but it decreased at the rate of 5.180 RMB per month after the implementation of the FCEMs policy (p = 0.006). The NRCMS reimbursement rate increased at 0.066% per month in pre-implementation of policy and the increasing trend was sharper at 0.349% in post-implementation of policy (p = 0.135). The daily hospitalization costs per patient decreased by 6.263 RMB (p = 0.030) per month, whereas it increased at the rate of 3.119 RMB (p = 0.002) per month afterwards. Conclusions: Based on interrupted time series analyses, we concluded that FCEMs policy was associated with positive changes of average LOS and average OOP expenses. The FCEMs policy has alleviated the financial burden of the rural seniors and slightly improved the efficiency of primary health service utilization. However, it had no positive effect on daily hospitalization costs. Therefore, in the general framework of FCEMs policy, the Chinese health policy-maker should take necessary supporting measures to curb climbing hospitalization expenditures and promote the rational drug use in primary healthcare institutions.

1. Introduction

In 2017, a census by China′s Statistics Bureau reported that the number of people aged 60 years and older has risen to 241 million, or approximately 17.3% of the total population of China [1]. It is predicted that the Chinese aging population will grow more than threefold over the next several decades. If this progress continues, the United Nations (UN) estimates that aging population will reach to 400 million and account for one third of the national population in 2030 [2]. With the growth of the geriatric population, China is facing the increasing aging-society challenge. Particularly, increases of numbers of seniors in the population coupled with the increased chronic disease burden associated with aging will provide the Chinese healthcare system with a huge challenge of delivering cost-effective primary healthcare service [3]. Due to an increase in chronic diseases, such as diabetes, hypertension, and chronic obstructive pulmonary disease, there is a growing demand for continuous-use medicines to treat such diseases. Meanwhile, the basic medical insurances have covered more than 95% of the population, while the low level of quality care is unable to address the concerns that elderly with prevalent diseases, especially for rural inpatients. It has also been demonstrated that the serious burden for patients is attributable to escalating medical costs for the treatment of a few chronic conditions in the primary care setting [4,5].
A study conducted in western rural China revealed that high out-of-pocket (OOP) spending and low healthcare reimbursement rate led to medical impoverishment, which was an important influencing factor to poverty [6]. On the other hand, drug expenditure accounted for 42% of the total expenditure for public hospitals on average in 2009 [7]. In primary healthcare institutions, about two thirds of outpatient expenditure and half of inpatient expenditure were sourced from drugs. Moreover, in the past decades, drug expenditure continued to increase at an average annual rate of 14% [8]. With the increased drug expenditure, the financial burden of chronic diseases for the elderly grew rapidly. In order to slow down health expenditure growth, China carried out a new round of healthcare system reforms since 2009. National Essential Medicine Policy (NEMP), as one of the key components in new healthcare reform, was established to improve rational use of drugs and promote accessibility and affordability of medicines, and reduce the financial burden of patients. The concept of essential medicines proposed by the World Health Organization (WHO) referred to those who can satisfy priority health care needs of the population [9]. Although the implementation of the NEMP in primary health care facilities has been progressing, the availability of essential medicines is generally low and some essential medicines are unaffordable in China [10,11,12]. Less equitable access to essential medicines has been a formidable public health challenge in China, particularly for rural areas [13].
The full coverage of essential medicines (FCEMs) policy has been put forward to improve the availability and affordability of essential medicines worldwide. In accordance with a report of the Pharmaceutical Country Profiles (PCP), out of 105 countries of the WHO, 54 countries implemented the policy to promote free access to essential medicines [14,15]. In different countries, the FCEMs policy can be adopted in different forms such as essential medicines at no charge or copayment exemption for essential medicines. Recently, the effects of free access to medicines among the elderly people has been the subject of some empirical studies in Western countries. For instance, Puig-Junoy et al. evaluated the effects of coinsurance exemption for prescription drugs applied to the elderly after retirement in Spain and found that the zero copayment increased the prescription medicines expenditure on average by 17.5%, pharmaceutical spending by 25% and the costs paid by the insurer by 60.4%. However, the work did not find a significant decline in hospitalization rates for elderly people exempted from copayment [16]. Paniz VMV et al. examined the effects of free access to medicines for 4003 elderly inhabitants living in 41 Brazilian cities and proved that the free continuous-use medicines programs played a positive role in increasing availability of medicines among the elderly with chronic diseases [17]. In China, a series of FCEMs policies for psychosis, chronic diseases, infectious diseases and child vaccines were carried out since ten years ago. However, only a few studies investigated the effects of FCEMs policy. Most of these studies focused on the outcome indicators related with medicines at no charge directly, such as medication adherence or drug expenses [18,19,20]. These observational studies suggest that the full coverage strategy increases medication adherence of patients with chronic conditions [18,19,20], but its effects on actual healthcare spending and utilization have not been rigorously assessed. Chinese literature paid little attention to the effects of FCEMs policy on utilization and accessibility of the healthcare service for rural seniors. Usually, most research on the effects of FCEMs policy only adopted the pre–post design and incorporated short-term data in certain hospitals. Furthermore, few studies seemed to have commented whether the FCEMs policy benefits the rural elderly population by analyzing aggregate medical insurance data. More research works to assess the long-term effects of FCEMs policy is needed.
Qidong is a rural Chinese county, which is located in eastern Jiangsu province. Since 2015, the government of Qidong implemented the FCEMs policy for rural seniors aged 70 and older. Therefore, as an empirical study, the objectives of present study were: (1) to detect the long-term effect of FCEMs policy on primary healthcare service utilization and (2) to assess the long-term effect of FCEMs policy on hospitalization costs and economic burden of diseases in rural elderly. As FCEMs policy was implemented in primary healthcare institutions of Qidong County, rural seniors who enrolled in New Rural Cooperative Medical System (NRCMS) were the target population.

2. Materials

2.1. Settings

The study was conducted in Qidong County in Jiangsu Province, China. Qidong is located to the north of Shanghai, at the Yangtze River Estuary, on the east coast of China. It has a coastal area of 1208 km2, and had a population of 1.12 million at the end of 2016. With the elderly population accounting for 23.01% of the total population since 2014, Qidong County has faced with the challenge of aging population. In Qidong County, there were 43 primary health institutions in total. Approximately, 4.6 medical technicians and 4.17 hospital beds for every one thousand inhabitants are available in this county. Healthcare service was primarily provided by 2207 licensed physicians and 1938 registered nurses in 2017 [21].

2.2. FCEMs Policy for Rural Seniors in Qidong County

From the start of NEMP in Qidong County in 2009, all the essential medicines were included in the reimbursement directory of NRCMS. In order to further eliminate out-of-pocket (OOP) costs for elderly patients, the FCEMs policy was implemented since January 2015. The essential medicines covered free of charge were selected from National Essential Medicine List (NEML) based on the following standard, which is as follows: (1) the retail price of oral dosage forms was less than 20 RMB and (2) the retail prices of injection dosage forms was less than 10 RMB. According to the FCEMs policy documents that were issued by the government of Qidong County, only the rural seniors aged 70 and older were the target beneficiaries of the policy. As a result, the subjects of this study were limited to those elderly patients over 70 years old. For the elderly hospitalized patients aged over 70 years at primary health institutions, NRCMS provided full reimbursement proportion for these essential medicines. According to statistics of the local health authority, the consumption of these medicines accounted for more than 80% of the total consumption of essential medicines in primary health institutions between 2014 and 2017.

2.3. Study Outcomes

We conducted an interrupted time series study, and the impacts of FCEMS policy on utilization and accessibility of primary healthcare service were evaluated for the elderly beneficiaries. Five outcome indicators were analyzed: number of hospitalizations, average length of stay (LOS), daily hospitalization costs per patient, average OOP expenses and NRCMS reimbursement rate (Table 1). Generally, the number of hospitalizations and LOS are measured as indicators of the efficient use of health service. Significantly reduced LOS and increased admissions could be associated with greater healthcare service utilization [22]. Daily hospitalization costs per patient, OOP expenses and NRCMS reimbursement rate were selected as indicators for accessibility of healthcare service [23]. For these indicators, the related intervention was the implementation of FCEMs policy, which tried to reduce the health expenditure and out-of-pocket expenses and increase medical insurance reimbursements.
These five outcome variables are aggregated monthly data at the patient level. The number of hospitalizations was measured by the hospital admissions in a given month. LOS was measured by the average hospital bed-days of the elderly patients hospitalized in a given month. The hospitalization costs were calculated where total expenses for beds, drugs and medical examinations incurred throughout the hospitalization period. Daily hospitalization costs per patient were calculated by using the overall inpatient expenses divided by number of hospitalizations and then divided by the average LOS during a given month. OOP expenses referred to patient payments for deductibles, as well as cash outlays for the items not covered by NRCMS. Average OOP expenses were measured by using total hospitalization costs minus expenses covered by NRCMS per patient in a given month. Similarly, average NRCMS reimbursement rate was computed by using expenses covered by NRCMS divided by total hospitalization costs per patient in a given month.

2.4. Data Collection

In Qidong County, from 2014 to 2017, there were 43 primary health institutions, which included 7 rural community health service centers and 36 township hospitals. All of the 43 primary healthcare institutions in Qidong County were included in this study. Meanwhile, all of the raw NRCMS’ claims data on the inpatients aged 70 years or older in the 43 primary health institutions was retrieved from the database of the department of the new rural medical scheme in Qidong County. Monthly data from January 2014 to November 2017 were collected. The data analysis focused on the elderly patients’ demographic characteristics (sex and age), hospitalization information (diagnosis, admission date, discharge date, number of hospitalizations and hospitalization costs) and reimbursement status (OOP expenses and reimbursement of NRCMS). Totally, data related to 48,071 patients and 91,444 patient admissions were obtained, which included the following variables: sex, age, health institutions, NRCMS ID, diagnosis, date of admission to hospital, date of discharge from hospital, total hospitalization costs, total reimbursement of NRCMS and total OOP expenses. The average age of the patients was 79.31 (Standard Deviation = 6.18). Less than two thirds of patients (59.9%) were female. The data for average LOS, daily hospitalization costs per patient, average OOP expenses and NRCMS reimbursement rate were based on these patients.
Approval for this study was obtained from the Institutional Review Board of Nanjing Medical University.

2.5. Statistical Analysis

An interrupted time series (ITS) regression was employed to analyze the 47-month data. When it is difficult or impossible to find a control group, the ITS model is regarded as the quasi-experimental design to analyze the longitudinal effects of interventions [24,25,26,27]. By using ITS in statistical terms, we can assess whether an intervention led to a transient or long-term outcome [28]. In this ITS model, in order to provide independent tests, the variation within the data was segmented into three parts: (1) the slope in scores for the pre-FCEMs policy period, (2) the change in level for the post-FCEMs policy period, accounting for the pre-FCEMs trend, and (3) the change in slope from pre-FCEMs policy to post-FCEMs policy. Given that the implementation of FCEMs policy in January 2015, hence, we set this month as the beginning of policy intervention. The pre-FCEMs policy segment of ITS analysis was fit with an intercept and a variable estimating trend [29], then trends of outcomes changing in slope were monitored in post-FCEMs policy period and changes in level were observed from pre-FCEMs policy period to post-FCEMs policy period. The ITS model was formulated as following:
Y t = β 0 + β 1 T + β 2 D + β 3 P + ε
In detail, Y is the main outcome indicator. T is a count variable indicating the time in months throughout the survey, assigning 1 to 47 in this study; D is a dummy variable representing the period before and after the FCEMS policy intervention, in which D = 0 for pre-FCEMS policy period and D = 1 for post-FCEMS policy period; P is a continuous variable counting the number of months after the FCEMs policy intervention, for the time before the FCEMs policy intervention P = 0. Furthermore, β 0 indicates the baseline level of the outcomes when T = 0; β 1 estimates the change of trends prior to the FCEMS policy; β 2 is the instantaneous level change in the implementation of the policy and β 3 is interpreted as the change in slope of trend in the outcome before and after FCEMS policy. Hereby, β 1 + β 3 is denoted as the actual trend of the outcomes after the implementation, indicating the net effect of policy intervention. ε is the error term. The ITS model was shown in Figure 1.
We employed Augment Dickey-fuller (ADF) unit root test to identify the stationarity of series data [29]. The null hypothesis of ADF test is that there is a unit root at some level of confidence, which suggests non-stationarity of the time series data. The results of ADF statistics are negative numbers. The more negative, the more likely a rejection of the null hypothesis. In this study, if the statistics of the five indicators were less than 5% critical value, we could conclude that these time series data were stationary. Meanwhile, the Durbin Watson test was employed to test the first-order autocorrelation of the data [30]. It is accredited that when the Durbin Watson statistics are approaching 2, it is extremely possible the observations are independent. The feasible generalized least square (FLGS) method would be employed to modify the first-order autocorrelation errors [31,32,33]. Analyses were performed with STATA v.14 software (STATA Corporation, College Station, TX, USA) and a statistical significance level of 0.05 was set for rejecting the null hypothesis.

3. Results

The annual five indicators from January 2014 to November 2017 were shown in Table 2. The results showed an increasing trend on the daily hospitalization costs per patient during the study period. Meanwhile, the results showed a decreasing trend on average LOS and average OOP expenses, while there existed an opposite trend on NRCMS reimbursement rate.
Table 3 showed the results of the ADF unit root test. The statistics of five indicators were less than 5% critical value and suggested these series data were stationary (p < 0.05). As shown in Table 4, all of the Durbin-Watson statistics are approaching 2. Due to the corrections by FLGS method, there were no autocorrelations in the observations.

3.1. Primary Health Service Utilization

As Table 4 shows, a declining trend was found in the number of hospitalizations from month to month before the FCEMs policy, but this decrease was not significant (p = 0.227). After the FCEMs policy started, no abrupt change in level was noted, but change from decreasing to increasing in the trend can be observed (p = 0.034). The change in slope also showed an increase but not statistically (p = 0.238).
As shown in Table 4, the results of parameter estimation of ITS analysis on average LOS after the correction for autocorrelation and Figure 2b showed the time series of average LOS of the elderly in primary health institutions. The average LOS showed an increasing trend and the rate of the increase was 0.372 (p < 0.001) bed days per month prior to the implementation of the FCEMs policy, then the average LOS decreased by 3.257 bed days immediately (p < 0.001) at the beginning month of intervention. In post-FCEMs policy period, the upward trend of average LOS was slowed down and the rate of increase was 0.003 bed days per month (p < 0.001).

3.2. Daily Hospitalization Costs of the Elderly Inpatients

The results of the parameters estimation of daily hospitalization costs per patient were shown in Table 4 and time series of hospitalization costs per patient was illustrated in Figure 2b. Daily hospitalization costs per patient decreased at the rate of 6.263 RMB (p = 0.030) per month before the implementation of FCEMs policy. At the beginning month of policy intervention, an increase in level was observed and daily hospitalization costs per patient increased by 13.844 RMB (p = 0.523) immediately. However, the daily hospitalization costs per patient increased at the rate of 3.119 RMB (p = 0.002) afterwards.

3.3. OOP Expenses and NRCMS Reimbursement Rate

Table 4 showed the results of the financial burden of the elderly estimated by ITS model during the study period and Figure 2 illustrated the overall trend of NRCMS reimbursement rate and average OOP expenses. The NRCMS reimbursement rate increased at 0.066% per month in pre-FCEMs policy period and the increasing trend was sharper at 0.349% in post-FCEMs policy (p = 0.135). Average OOP expenses showed the opposite trend before and after the policy intervention. At first it increased steadily at the rate of 38.035 RMB per month (p = 0.012) prior to the intervention, and then an immediate drop in level was observed and the average OOP expenses descended by 626.905 RMB (p < 0.001) at the beginning month of policy intervention. However, the average OOP expenses decreased by 5.18 RMB per month (p = 0.006) after the implementation of FCEMs policy.

4. Discussion

Using 47-month series medical insurance data from January 2014 to November 2017, an interrupted time series analysis in this study evaluated the impacts of FCEMs policy on number of hospitalizations, average LOS, daily hospitalization costs per patient and patient copayments. The information provided in this work demonstrated the policy resulted in a decrease of the OOP expenses and an increase in daily hospitalization costs per patient, while contributed to a deceleration of the growth of LOS. Our results also suggested that the implementation of FCEMs policy in Qidong County had a slight effect on increasing number of hospitalizations and NRCMS reimbursement rate.
In this study, the number of hospitalizations and average LOS were employed to assess the efficiency of health care utilization, evidently, a shorter LOS has been associated with an increase of hospital expenses [34]. Due to free access to essential medicines in elderly hospitalized patients, we assumed that hospitalizations and LOS increased rapidly after FCEMs policy. As is expected, after the month of FCEMs policy, a significant transient rise in number of hospitalizations was observed. However, the policy generated little or no impact on increasing the number of hospitalizations. Meanwhile, two different findings were observed by the descriptive analysis and the ITS model, respectively. The findings of the ITS model showed that the FCEMs policy resulted in a deceleration of the growth of LOS in Qidong County. However, surprisingly, contrary to our hypothesis, the average LOS per month was reduced after the implementation of FCEMs policy based on the results of descriptive analysis. As the elderly outpatients were excluded from the beneficiary population, the beneficiaries of FCEMs policy were only the elderly inpatients. Compared with usual access to medicines, the elderly inpatients could take the essential medicines at no charge. Two potential reasons could be proposed to explain this finding. First, The FCEMs policy could enhance aged patients′ compliance with drugs and promote the efficiency of essential medicines in the treatment of chronic diseases. Some studies in Western countries claimed the free access to medicines contributed to a positive impact on medication adherence. For example, a study from the United States confirmed that the medication adherence of patients with myocardial infraction had been improved as a result of full coverage for preventive medications [35]. In general, improvements in medication adherence usually led to the achievement of longer-term therapeutic and outcome goals [34,35,36,37], especially for reductions in LOS [36,37]. Therefore, the deceleration of the growth of LOS could be associated with the implementation of FCEMS policy. Second, the LOS of patients with mild-to-moderate illness severity could be shorter than the patients with severe illness. Some previous empirical studies in China could demonstrate the relationship between the patients′ disease severity and LOS. For example, Li et al. demonstrated that the disease severity was the most important influencing factor for length of stay, and the more severe the disease of patients, the longer the length of stay [38]. Mo et al. also found that the disease severity had an association with length of stay [39]. According to our data analysis in this study, no significant change was observed in the elderly patients′ disease category before and after the FCEMs policy. Under the premise of reduction of LOS after the FCEMs policy, we can conclude that there were changes in the disease severity of elderly inpatients during the pre- and post-FCEMs policy period. After the implementation of FCEMs policy, in order to get essential medicines at no charge, more and more patients with mild-to-moderate illness severity were transferred to the inpatient units from outpatient clinics in primary health care institutions. Compared with the patients with severe illness, the LOS of the patients with mild-to-moderate illness severity could be shorter. Therefore, the changes in the disease severity of elderly inpatients led to reduction of LOS after the FCEMs policy.
The average OOP expenses increased before the implementation of the FCEMs policy, but it decreased afterwards. The finding of the impact of FCEMs policy on OOP expenses was similar with previous studies that claimed a reduction in OOP expenses owing to dispensing drugs free of charge [35,40,41]. It is considered that improved reimbursement rate driven by the FCEMs policy has the potential to save medical costs and reduce the financial burden of the elderly. For example, a study investigating patients with early breast cancer found that the full coverage of aromatase inhibitors had yielded greater health benefits at a lower cost [34]. In addition, another study concentrating on post-myocardial infraction patients suggested the full prescription drug coverage was able to save money over the long-term [42]. Furthermore, from a society perspective, this finding is encouraging as it is likely to improve health care equities and decrease the incidence of medical impoverishment. For instance, Xu et al. suggested that the provision of drugs at free charge to patients with hypertension resulted in reduced medical costs and financial burden, along with improved accessibility and equity of drugs [43]. While a study focused on chronic disease treatment indicated free access to essential medicines has been effective to the poor population and decreased inequalities, but with differences in regions and classes of medicines [44].
Another key finding of our study is that the NRCMS reimbursement rate had been increasing from 53.29% to 70.35% throughout the study period in Qidong County. NRCMS reimbursement rate was used to evaluate the accessibility of primary health service in this study. Usually, a higher health insurance reimbursement rate contributed to an improvement of accessibility of health service [45,46]. On one hand, the increasing NRCMS reimbursement rate indicated that FCEMs policy improved the accessibility of health service for rural seniors. Evidence demonstrated patients with lower reimbursement rate were less likely to receive healthcare service [47]. Due to the increasing NRCMS reimbursement rate, the expenses of health service were more affordable for the patients, who could stimulate the willingness of patients to seek more health service [48], particularly for the rural senior beneficiaries with low-income and low-health [49]. On the other hand, the increasing NRCMS reimbursement rate had the potential to lead to an unreasonable increase in reimbursement expenditure. Simultaneously, the reimbursement for other health-insurance beneficiaries could be restricted due to limited health care funding.
However, the FCEMs policy also resulted in a rapid growth of daily hospitalization costs per patient in Qidong County. Although the daily hospitalization costs per patient decreased before the implementation of FCEMs policy, it increased after the implementation of FCEMs policy. The positive relationship between health expenditure and reimbursement rate is well-established in previous literature [50]. We have hypothesized that the hospitalization costs could increase dramatically after FCEMs policy. Coincidentally, this finding suggested that the daily hospitalization costs per patient increased after the implementation of FCEMs policy. A complex set of policy, healthcare, technology, and economic factors can result in escalation of the daily hospitalization costs and it is unlikely to be addressed by FCEMs policy alone. One possible reason is that the FCEMs policy unleashed the rural seniors′ pent-up health demand in primary health care institutions. In general, the FCEMs policy had no positive effect on hospitalization costs. Empirical evidence indicated the irrational use of medical service will abound if there is a full coverage of health care. A national survey data indicated that health insurance coverage led to a significant effect on the possibility that the elderly would overuse health service [51]. From the perspective of service providers, to compensate for the loss due to decreased drug costs, health institutions have the possibility to increase expenses of treatment and examination, while physicians may induce patients, especially those with rural medical insurance, to receive excessive medical treatment [52]. Therefore, constant efforts are required to reduce unreasonable health expenses in Qidong County. First, the criterion for hospitalization should be established to control the rapid increase in unreasonable admissions, along with strengthened supervision of health care-seeking behaviors, and the physicians should be responsible for accurate suggestions on seeking service to avoid elderly patients taking up medical resources excessively [53]. Second, considering repetitive distribution over a short period to the elderly may cause waste of essential medicines, the frequency of distribution and classes of essential medicines need to be regulated effectively.
Meanwhile, as the strongest, quasi-experimental design [54,55], interrupted time series analysis was employed to evaluate the long-term effects of the FCEMs policy. Nevertheless, our study is subject to several other limitations. First, it lacks a control group to explore whether there are other factors, rather than the policy implementation, that result in the change of primary outcomes. Although it is not required for ITS analysis to establish associations between interventions and outcomes, a control group can help us better understand the impact of the interventions. Second, as a result of constraint of data, the outcomes in the study are not so comprehensive that we cannot assess the impact of the policy on medication adherence, clinical outcomes, irrational use of essential medicines and health equities. Third, limited to the setting in a county, the study should have been carried out in more representative municipalities, provinces and countries ideally. However, the FCEMs policy in infancy has no national criteria for selection and distribution of essential medicines and the beneficiaries depend on the decision of the local governments. Fourth, due to the local policy provisions, there is a restriction on age of beneficiaries in this study. Only the elderly inpatients aged 70 years or older are selected as the target population, limiting generalizability to the entire healthcare systems.

5. Conclusions

In summary, this study analyzed the changes in the number of hospitalizations, average LOS, daily hospitalization costs per patient, average OOP expenses and NRCMS reimbursement rate before and after the implementation of FCEMS policy in Qidong County of Jiangsu Province, Eastern China during the 47-month period from 2014 to 2017. Based on the interrupted time series analysis, we concluded that FCEMs policy was associated with positive changes of LOS. The FCEMs policy decelerated the growth of LOS and reduced average LOS per month simultaneously. Meanwhile, the FCEMs policy implemented at Qidong County seemed to have positive impacts on average OOP expenses and NRCMs reimbursement rate. It has alleviated the financial burden of the rural seniors and slightly improved the efficiency of primary health service utilization. However, the FCEMs policy had no positive effect on daily hospitalization costs. As a consequence, effective initiatives should be taken such as strengthen the supervision, improve rigorous selection and distribution of essential medicines at free charge to constrain climbing health spending.

Author Contributions

Conceptualization, Y.W. and X.L.; Data curation, Y.W., Y.Z., H.S., X.S. and X.L.; Formal analysis, Y.W. and H.S.; Funding acquisition, N.C. and X.L.; Investigation, Y.W., Y.Z., H.S., X.S., N.C. and X.L.; Methodology, Y.Z., N.C. and X.L.; Project administration, Y.W. and X.L.; Writing—Original draft, Y.W. and X.L.; Writing—Review & editing, X.L.

Funding

This research was supported by the grant of National Natural Science Foundation of China (Grant No. 71673147), the General Project of Philosophy and Social Science of University of Jiangsu Province (Grant No: 2017SJB0277), the Joint Project between Southeast University and Nanjing Medical University (Grant No: 2018DN0023) and National Social Science Foundation of China (Grant No:15CRK015).

Acknowledgments

Special thanks to Xinhua Sun from the Qidong Commission of Health for his help in data collection.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. National Bureau of Statistics of China. China Statistical Yearbook 2018; Statistics Press: Beijing, China, 2018.
  2. United Nations Department of Economic and Social Affairs. World Population Prospects: The 2010 Revision; United Nations: New York, NY, USA, 2011. [Google Scholar]
  3. Pearson, W.S.; Bhat-Schelbert, K.; Probst, J.C. Multiple chronic conditions and the aging of America: Challenge for primary care physicians. J. Prim. Care Community Health 2012, 3, 51–56. [Google Scholar] [CrossRef] [PubMed]
  4. Qiu, Y.; Xu, W.; Cao, S.; Gao, Q.; Wang, Y. Study on disease burden of elderly residents in rural areas. Chin. J. PHM 2014, 30, 143–146. (In Chinese) [Google Scholar]
  5. Qian, J.; Chen, Y.; Meng, Q. Trends analysis and strategic measures for diseases economic burden of the aging population in China. Chin. J. Health Policy 2012, 5, 12–16. (In Chinese) [Google Scholar]
  6. Wang, H.; Wang, Z.; Ma, P. Analysis and reflection on poverty caused by illness in rural areas based on the survey of 1214 poor households caused by illness in 9 western provinces and cities. Economist 2016, 10, 71–81. (In Chinese) [Google Scholar]
  7. Chinese Ministry of Health. 2010 Chinese Health Statistical Digest; Chinese Ministry of Health: Beijing, China, 2011.
  8. National Health Development Research Center (NHDRC). China National Health Accounts Report; National Health Development Research Center (NHDRC): Beijing, China, 2013. [Google Scholar]
  9. Guo, Z.; Guan, X.; Shi, L. The impacts of implementation of National Essential Medicines Policies on primary healthcare institutions: A cross-sectional study in China. BMC Health Serv. Res. 2017, 17, 723. [Google Scholar] [CrossRef]
  10. Guan, X.; Liang, H.; Xue, Y.; Shi, L. An analysis of China’s national essential medicines policy. J. Public Health Policy 2011, 32, 305–319. [Google Scholar] [CrossRef]
  11. Fang, Y.; Wagner, A.K.; Yang, S.; Jiang, M.; Zhang, F.; Ross-Degnan, D. Access to affordable medicines after health reform: Evidence from two cross-sectional surveys in Shaanxi Province, western China. Lancet Glob. Health 2013, 1, e227–e237. [Google Scholar] [CrossRef]
  12. Shang, J.; Guo, Z.; Lin, Q.; Li, L.; Chen, C.; Feng, L.; Song, R.; Guan, X.; Shi, L. Analysis of accessibility of essential medicine in Beijing. Chin. J. Health Policy 2016, 2, 52–58. (In Chinese) [Google Scholar]
  13. Guan, X.; Wang, B.; Xin, X.; Guo, Z.; Han, S. Free pharmaceutical policy to promote equity and access: Its implications for China. Chin. J. Health Policy 2015, 2, 54–59. (In Chinese) [Google Scholar]
  14. Mudur, G. India’s health assurance plan will offer free medicines, diagnostics, insurance, and traditional medicine. BMJ 2014, 349, g6023. [Google Scholar] [CrossRef]
  15. Honda, A.; Hanson, K. Do equity funds protect the poor? Case studies from north-western Madagascar. Health Policy Plan. 2013, 28, 75–89. [Google Scholar] [CrossRef] [PubMed]
  16. Puig-Junoy, J.; García-Gómez, P.; Casado-Marín, D. Free medicines thanks to retirement: Impact of coinsurance exemption on pharmaceutical expenditures and hospitalization offsets in a national health service. Health Econ. 2016, 25, 750–767. [Google Scholar] [CrossRef] [PubMed]
  17. Paniz, V.M.; Fassa, A.G.; Facchini, L.A.; Picchini, L.A.; Tomasi, E.; Thume, E.; da Silverira, D.S.; Rodrigues, M.A.; Domingues, M.R.; Bertoldi, A.D. Free access to hypertension and diabetes medicines among the elderly: A reality yet to be constructed. Cadernos de Saúde Pública 2010, 26, 1163–1174. [Google Scholar] [CrossRef] [PubMed]
  18. Wu, T.; Qin, X.; Xu, Q.; Ma, J. Comparison of effect of treating essential hypertension between free of charge and self-paying in rural communities. Chin. J. School Dr. 2014, 28, 122–123. (In Chinese) [Google Scholar]
  19. Li, K.; Wang, Q.; Du, G.; Ma, T. The management effect of free medication for elderly hypertension patients in rural areas. Chin. J. Prev. Contr. Chron Dis. 2015, 23, 532–533. (In Chinese) [Google Scholar]
  20. Xu, A.; Bu, X.; Dong, L.; Ge, M.; Liu, P.; Liu, W. Effect evaluation of the free medication for poor people with serious mental illness in Weifang. J. Psychiatry. 2013, 26, 178–180. (In Chinses) [Google Scholar]
  21. National Bureau of Statistics. China Statistical Yearbook; China Statistics Publishing House: Beijing, China, 2014.
  22. Lin, C.; Ma, T.; Lin, C.; Kao, C. The impact of global budgeting on health service utilization, health care expenditures, and quality of care among patients with pneumonia in Taiwan. Eur. J. Clin. Microbiol. Infect. Dis. 2016, 35, 219–225. [Google Scholar] [CrossRef]
  23. Liu, C.; Zhao, J. The new rural cooperative medical service system and evaluation and its influencing factors: An empirical analysis based on data CHARLS. Econ. Probl. 2016, 2, 86–91. (In Chinese) [Google Scholar]
  24. Campbell, S.M.; Reeves, D.; Kontopantelis, E.; Sibbald, B.; Roland, M. Effects of pay for performance on the quality of primary care in England. N. Engl. J. Med. 2009, 361, 368–378. [Google Scholar] [CrossRef]
  25. Serumaga, B.; Ross-Degnan, D.; Avery, A.J.; Elliott, R.; Majumdar, S.R.; Zhang, F.; Soumerai, S.B. Effect of pay for performance on the management and outcomes of hypertension in the United Kingdom: Interrupted time series study. BMJ 2011, 342, d108. [Google Scholar] [CrossRef]
  26. Wager, A.K.; Ross-Degnan, D.; Gurwitz, J.; Zhang, F.; Gilden, D.; Soumerai, S.B. Effect of medicaid of New York state regulatory action on benzodiazepine prescribing and hip fracture rates. Ann. Intern. Med. 2007, 146, 96–103. [Google Scholar] [CrossRef] [PubMed]
  27. Singer, J.D.; Willett, J.B. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence; Oxford University Press: New York, NY, USA, 2003. [Google Scholar]
  28. Wagner, A.K.; Soumerai, S.B.; Zhang, F.; Ross-Degnan, D. Segmented regression analysis of interrupted time series studies in medication use research. J. Clin. Pharm. Ther. 2002, 27, 299–309. [Google Scholar] [CrossRef] [PubMed]
  29. Emmerick, I.C.M.; Campos, M.R.; Luiza, V.L.; Chaves, L.A.; Bertoldi, A.D.; Ross-Degan, D. Retrospective interrupted time series examining hypertension and diabetes medicines usage following changes in patient cost sharing in the ‘Farmácia Popular’ programme in Brazil. BMJ Open 2017, 7, e017308. [Google Scholar] [CrossRef] [PubMed]
  30. Crosbie, J. Interrupted time series analysis with brief single-subject data. J. Consult. Clin. Psychol. 1993, 61, 966–974. [Google Scholar] [CrossRef]
  31. Orcutt, H. Application of least squares regression to relationships containing autocorrelated error terms. J. Am. Stat. Assoc. 1949, 44, 32–61. [Google Scholar]
  32. Choi, C.Y.; Hu, L.; Ogaki, M. Robust estimation for structural spurious regressions and a hausman-type cointegration Test. J. Econom. 2008, 142, 327–351. [Google Scholar] [CrossRef]
  33. Ali, S.; Liu, Y.; Ishaq, M.; Shah, T.; Ilyas, A.; Din, I. Climate change and its impact on the yield of major food crops: Evidence from Pakistan. Foods 2017, 6, 39. [Google Scholar] [CrossRef]
  34. Healy, W.L.; Rana, A.J.; Iorio, R. Hospital economics of primary total knee arthroplasty at a teaching hospital. Clin. Orthop. Relat. Res. 2011, 469, 87–94. [Google Scholar] [CrossRef]
  35. Ito, K.; Avorn, J.; Shrank, W.H.; Toscano, M.; Brennan, T.; Choudhry, N.K. Long-Term cost-effectiveness of providing full coverage for preventive medications after myocardial infarction. Circ. Cardiovasc. Qual. Outcomes 2015, 8, 252–259. [Google Scholar] [CrossRef]
  36. Curtis, S.E.; Boye, K.S.; Lage, M.J.; Garcia-Perze, L.E. Medication adherence and improved outcomes among patients with type 2 diabetes. Am. J. Manag. Care 2017, 23, e208–e214. [Google Scholar]
  37. Hood, S.R.; Giazzon, A.J.; Seamon, G.; Lane, K.A.; Wang, J.; Eckert, G.J.; Tu, W.; Murray, M.D. Association between medication adherence and the outcomes of heart failure. Pharmacotherapy 2018, 38, 539–545. [Google Scholar] [CrossRef] [PubMed]
  38. Li, H.; Bu, W.; Zhang, X.; Xu, F.; Wang, D. Analysis of influence factors of average length of stay in the general third class comprehensive hospital. Chin. Hosp. Manag. 2014, 34, 41–43. (In Chinese) [Google Scholar]
  39. Mo, C.; Yang, T.; Shi, Y. Research on average length of stay as an efficient index for comparing inter-hospital evaluation. Chin. Hosp. 2009, 13, 16–18. (In Chinese) [Google Scholar]
  40. Yu, B.; Zhang, X.; Wang, G. Full coverage for hypertension drugs in rural communities in China. Am. J. Manag. Care 2013, 19, e22–e29. [Google Scholar]
  41. Felix, K.; Lutz, H.; Maarten, P. Availability and affordability of antimalarial and antibiotic medicines in Malawi. PLoS ONE 2017, 12, e0175399. [Google Scholar]
  42. Choudhry, N.K.; Patrick, A.R.; Antman, E.M.; Avorn, J.; Shrank, W.H. Cost-Effectiveness of providing full drug coverage to increase medication adherence in post-myocardial infarction medicare beneficiaries. Circulation 2008, 117, 1261–1268. [Google Scholar] [CrossRef]
  43. Xu, Y.; Yang, H.; Bo, Y.; Chen, H.; Cai, Y. A pilot research on operational mode of free hypertensive drug administration in community. Chin. J. Soc. Med. 2013, 30, 48–50. (In Chinese) [Google Scholar]
  44. Tavares, N.U.; Luiza, V.L.; Oliveira, M.A.; Costa, K.S.; Mengue, S.S.; Arrais, P.S.; Ramos, L.R.; Farias, M.R.; Pizzol, T.D.; Bertoldi, A.D. Free access to medicines for the treatment of chronic diseases in Brazil. Rev. Saude Publica 2016, 50 (Suppl. 2), 7s. [Google Scholar] [CrossRef]
  45. Chalmers, N.I.; Compton, R.D. Children’s access to dental care affected by reimbursement rates, dentist density, and dentist participation in medicaid. Am. J. Public Health 2017, 107, e1–e3. [Google Scholar] [CrossRef]
  46. Xu, J.; Wang, J.; King, M.; Liu, R.; Yu, F.; Xing, J.; Su, L.; Lu, M. Rural–Urban disparities in the utilization of mental health inpatient services in China: The role of health insurance. Int. J. Health Econ. Manag. 2018, 18, 377–393. [Google Scholar] [CrossRef]
  47. Su, S.; Bao, H.; Wang, X.; Wang, Z.; Li, X.; Zhang, M.; Wang, J.; Jiang, H.; Wang, W.; Qu, S.; et al. The quality of invasive breast cancer care for low reimbursement rate patients: A retrospective study. In Proceedings of the China Health Statistics Annual Conference, Wuhan, China, 22–25 August 2017. [Google Scholar]
  48. Cai, C.; Li, L. Medical insurance and medical care demand for the elderly in China. Econ. Res. J. 2011, 46, 95–107. [Google Scholar]
  49. Yu, D. Does basic medical insurance for urban residents raise medical service utilization? An empirical evaluation of system running effect. Financ. Econ. Res. 2015, 30, 117–128. [Google Scholar]
  50. Li, J.; Feng, X.L. Health care-seeking behaviors and health expenditures in adults aged 45 years and older in China, 2011–2013. Trop. Med. Int. Health 2017, 22, 638–654. [Google Scholar] [CrossRef] [PubMed]
  51. Hu, H.; Luan, W.; Li, J. Medical insurance health services utilization and excessive demands for medical services—The impact of medical insurance on utilization of health service of the elderly. J. Shanxi Univ. Financ. Econ. 2015, 37, 14–24. (In Chinese) [Google Scholar]
  52. Yao, Y.; Liu, B.; Liu, G.; Zang, W. Medical insurance, household registration system and healthcare utilization: Evidences from CHARLS data Analysis. Insur. Stud. 2014, 6, 105–116. (In Chinese) [Google Scholar]
  53. Miao, Y.; Gu, J.; Zhang, L.; He, R.; Sandeep, S.; Wu, J. Improving the performance of social health insurance system through increasing outpatient expenditure reimbursement ratio: A quasi-experimental evaluation study from rural China. Int. J. Equity Health 2018, 17, 89. [Google Scholar] [CrossRef]
  54. Rashidian, A.; Joudaki, H.; Khodayari-Moez, E.; Omranikhoo, H.; Geraili, B.; Arab, M. The impact of rural health system reform on hospitalization rates in the Islamic Republic of Iran: An interrupted time series. Bull. World Health Organ. 2013, 91, 942–949. [Google Scholar] [CrossRef]
  55. Hyman, R. Quasi-Experimentation: Design and analysis issues for field settings (Book). J. Person. Assess. 1982, 46, 96–97. [Google Scholar] [CrossRef]
Ijerph 16 04316 g001 550
Figure 1. Graphic illustration of interrupted time series model and measuring the policy effects based on trend lines for data points before and after the policy.
Figure 1. Graphic illustration of interrupted time series model and measuring the policy effects based on trend lines for data points before and after the policy.
Ijerph 16 04316 g001
Ijerph 16 04316 g002 550
Figure 2. (a) Impacts of full coverage of essential (FCEMs) policy on average out-of-pocket (OOP) expenses and number of hospitalizations. (b) Impacts of FCEMs policy on average length of stay (LOS), New Rural Cooperative Medical System (NRCMS) reimbursement rate and daily hospitalization costs per patient day.
Figure 2. (a) Impacts of full coverage of essential (FCEMs) policy on average out-of-pocket (OOP) expenses and number of hospitalizations. (b) Impacts of FCEMs policy on average length of stay (LOS), New Rural Cooperative Medical System (NRCMS) reimbursement rate and daily hospitalization costs per patient day.
Ijerph 16 04316 g002
Table
Table 1. Study outcome variables.
Table 1. Study outcome variables.
OutcomesIndicatorsUnits of Measurement
Utilization of primary healthcare servicesNumber of hospitalizationsNumber of patients
Average LOS aHospital inpatient bed-days
Accessibility of Primary Healthcare ServicesDaily hospitalization costs per patient bRMB
Average OOP expenses cRMB
NRCMS reimbursement rate dPercentages
a Average length of stay = l e n g t h   o f   s t a y n u m b e r   o f   h o s p i t a l i z a i t o n s ; b Daily hospitalization costs per patient   = t o t a l   h o s p i t a l i z a t i o n   c o s t s n u m b e r   o f   h o s p i t a l i z a t i o n s ∗ a v e r a g e   l e n g t h   o f   s t a y ; c Average out-of-pocket expenses = t o t a l   o u t − o f − p o c k e t   e x p e n s e s n u m b e r   o f   h o s p i t a l i z a i t o n s ; d NRCMS reimbursement rate = t o t a l   r e i m b u r s e m e n t   o f   N R C M S t o t a l   h o s p i t a l i z a i t o n   c o s t s ; LOS, length of stay. OOP, out-of-pocket. NRCMS, new rural cooperative medical system. RMB, renminbi, Chinese currency.
Table
Table 2. Outcomes of the elderly in primary health institutions during study period in Qidong County, Jiangsu Province, 2014 to November 2017.
Table 2. Outcomes of the elderly in primary health institutions during study period in Qidong County, Jiangsu Province, 2014 to November 2017.
Indicators2014201520162017
Number of hospitalizations19,07624,09223,16925,109
Average LOS (bed-days)9.288.168.497.99
Daily hospitalization costs per patient, monthly average (RMB)434.10440.36456.74518.29
Average OOP expenses, monthly average (RMB)1829.861387.621353.751221.73
NRCMS reimbursement rate (%)53.2961.0964.8970.35
Note. LOS, length of stay. OOP, out-of-pocket. NRCMS, new rural cooperative medical system. RMB, renminbi, Chinese currency.
Table
Table 3. Results of Augment Dicky-Fuller unit root test and stationarity of outcomes.
Table 3. Results of Augment Dicky-Fuller unit root test and stationarity of outcomes.
IndicatorsIntercept and TrendADF StatisticP Value 5% Critical ValueStationarity
Number of hospitalizationsintercept−3.4090.011−2.944stationarity
Average LOStrend−6.839<0.001−2.941stationarity
Daily hospitalization costs per patienttrend−6.038<0.001−3.516stationarity
Average OOP expensestrend−3.4150.049−2.941stationarity
NRCMS reimbursement rateintercept and trend−3.5360.048−3.513stationarity
Note. ADF, Augment Dicky-Fuller. OOP, out-of-pocket. NRCMS, new rural cooperative medical system.
Table
Table 4. The changes in trend and level estimated by ITS regression analysis after correcting for autocorrelation.
Table 4. The changes in trend and level estimated by ITS regression analysis after correcting for autocorrelation.
IndicatorsSlope: Jan 2014 to Dec 2014 (SE)pChange in Level: Predicted Value of Jan 2015 Based on Post-FCEMs Trend Minus Predicted Value Based on Pre-FCEMs Trend (SE)pChange in Slope from Jan 2014-Dec 2014 to Jan 2015-Nov 2017 (SE)pParameters of Model Fit
DW Root MSER2
Number of hospitalizations−55.953 (45.602)0.227720.440 (329.545)0.03458.328 (48.691)0.2381.790351.4200.209
Average LOS0.372 (0.082)<0.001−3.257 (0.625)<0.001−0.369 (0.083)<0.0012.2671.1900.508
Daily hospitalization costs per patient−6.263 (2.793)0.03013.844 (21.478)0.5239.382 (2.841)0.0022.02835.7650.534
Average OOP expenses38.035 (11.555)0.012−626.905 (90.261)<0.001−43.215 (14.950)0.0061.891106.1300.741
NRCMS reimbursement rate0.066 (0.165)0.6925.301 (1.033)<0.0010.283 (0.185)0.1351.6361.0120.915
Note. FCEMs, full coverage of essential medicines. LOS, length of stay. OOP, out-of-pocket. NRCMS, new rural cooperative medical system. SE, standard error. Root MSE is the estimate of the standard deviation of residual error. R2 indicates the proportion of total variation ascribed to the model fit. The Durbin-Watson statistic lies between 0 and 4; generally, the closer the Durbin-Watson statistic is approaching 2, the stronger rejection of autocorrelation.

Share and Cite

MDPI and ACS Style

Wang, Y.; Zhu, Y.; Shi, H.; Sun, X.; Chen, N.; Li, X. The Effect of the Full Coverage of Essential Medicines Policy on Utilization and Accessibility of Primary Healthcare Service for Rural Seniors: A Time Series Study in Qidong, China. Int. J. Environ. Res. Public Health 2019, 16, 4316. https://doi.org/10.3390/ijerph16224316

AMA Style

Wang Y, Zhu Y, Shi H, Sun X, Chen N, Li X. The Effect of the Full Coverage of Essential Medicines Policy on Utilization and Accessibility of Primary Healthcare Service for Rural Seniors: A Time Series Study in Qidong, China. International Journal of Environmental Research and Public Health. 2019; 16(22):4316. https://doi.org/10.3390/ijerph16224316

Chicago/Turabian Style

Wang, Ying, Yulei Zhu, Hang Shi, Xiaoluan Sun, Na Chen, and Xin Li. 2019. "The Effect of the Full Coverage of Essential Medicines Policy on Utilization and Accessibility of Primary Healthcare Service for Rural Seniors: A Time Series Study in Qidong, China" International Journal of Environmental Research and Public Health 16, no. 22: 4316. https://doi.org/10.3390/ijerph16224316

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop