Next Article in Journal
Minority Identity, Othering-Based Stress, and Sexual Violence
Previous Article in Journal
Weekday and Weekend Differences in Eating Habits, Physical Activity and Screen Time Behavior among a Sample of Primary School Children: The “Seven Days for My Health” Project
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 7
10.3390/ijerph19074219
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Variability in Healthcare Expenditure According to the Stratification of Adjusted Morbidity Groups in the Canary Islands (Spain)

by
Maria Consuelo Company-Sancho
1,2,*,
Víctor M. González-Chordá
3 and
María Isabel Orts-Cortés
2,4,5
1
Health Promotion Service, Directorate General for Public Health, Canary Islands Health Service, 35003 Las Palmas de Gran Canaria, Spain
2
Nursing and Healthcare Research Unit (Investén-isciii), Institute of Health Carlos III, 28029 Madrid, Spain
3
Nursing Department, Universitat Jaume I, Avda Sos Baynat s/n, 12071 Castellon, Spain
4
Department of Nursing, University of Alicante (BALMIS), Alicante Institute for Health and Biomedical Research (ISABIAL), 03690 Alicante, Spain
5
CIBER of Frailty and Healthy Ageing, (CIBERFES) Institute of Health Carlos III, 28029 Madrid, Spain
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(7), 4219; https://doi.org/10.3390/ijerph19074219
Submission received: 5 March 2022 / Revised: 29 March 2022 / Accepted: 30 March 2022 / Published: 1 April 2022
(This article belongs to the Section Health Economics)
Browse Figure
Review Reports Versions Notes

Abstract

:
Morbidity is the main item in the distribution of expenditure on healthcare services. The Adjusted Morbidity Group (AMG) measures comorbidity and complexity and classifies the patient into mutually exclusive clinical categories. The aim of this study is to analyse the variability of healthcare expenditure on users with similar scores classified by the AMG. Observational analytical and retrospective study. Population: 1,691,075 subjects, from Canary Islands (Spain), aged over 15 years with data from health cards, clinical history, Basic Minimum Specialised Healthcare Data Set, AMG, hospital agreements information system and Electronic Prescriptions. A descriptive, bivariant (ANOVA coefficient η2) and multivariant analysis was conducted. There is a correlation between the costs and the weight of AMG (rho = 0.678) and the prescribed active ingredients (rho = 0.689), which is smaller with age and does not exist with the other variables. As for the influence of the AMG morbidity group on the total costs of the patient, the coefficient η2 (0.09) obtains a median effect in terms of the variability of expenditure, hence there is intra- and inter-group variability in the cost. In a first model created with all the variables and the cost, an explanatory power of 36.43% (R2 = 0.3643) was obtained; a second model that uses solely active ingredients, AMG weight, being female and a pensioner obtained an explanatory power of 36.4%. There is room for improvement in terms of predicting the expenditure.

1. Introduction

Appropriate management of chronic patients would improve health outcomes, boosting the sustainability of healthcare systems [1]. In Spain, the National Health System is financed with public funds through general taxation. The service is free, while prescribed pharmaceutical products entail co-payment by the patient. There are 17 autonomous regional communities, each with its own department of health that mainly controls the funding, organisation and provision of healthcare services in each territory, of which the Canary Islands are one.
The World Health Organization (WHO) defines comorbidity as the presence of two or more diseases that occur in a concomitant manner with a primary disease [2], with the possibility that the progress of both may worsen. Its prevalence increases progressively with age, between 60% and 70% among people over the age of 60 and 80 years, respectively [3]. It is a determining factor in accounting for the consumption of healthcare resources in chronicity [4], insofar as the more comorbidity there is, the greater the consumption of resources [5]. A tool that facilitates the analysis of this comorbidity and multi-morbidity will allow a better comprehension of the use, costs, and quality of the combination of services received in a population [4]. The strategies developed internationally and nationally to cope with chronicity propose risk stratification, which will help clinicians in their decision-making [6,7]
Stratifiers, also known as risk adjustment models, are a system for measuring morbidity, which classifies people into mutually exclusive clinical categories, based on information obtained from clinical histories. Taking comorbidity as the starting point, stratifiers introduce the concept of levels of complexity, attempting to predict the risk of occurrence of a given event or a certain level of resource consumption. Stratifiers use inputs, such as medical diagnoses, age, sex, contact with the healthcare system, surgical procedures or pharmaceutical prescription, to obtain scores that make it possible to classify the population into homogeneous morbidity groups. Usually, they do not take into account factors such as personal circumstances, social context, fragility, resilience to problems or nursing care, although care is regarded as a key element in strategies for dealing with chronicity [6].
There are different developments and their purpose is multifaceted, from identifying high-cost patients for case management, monitoring health status, determining patterns of resource use and consumption, or predicting population risk and cost, which is useful for pharmaceutical budget allocation. [8,9]. In short, they are all management tools.
In Spain, over the last 15 years, prolific research has been conducted on the functioning of ACGs and GRCs in some autonomous communities for various purposes, such as cost measurement [10,11], cost prediction [12], effectiveness of Primary Care (PC) referrals [13], such as morbidity stratification [14], analysis of excessively frequent use of services [15] and pharmaceutical dispensing [11,12,16,17]. This may be due to the rise in the phenomenon of chronicity, where healthcare systems have prioritised the detection of patients with greater complexity, in order to better manage services and resources. Adjusted Morbidity Groups (AMGs) are a risk adjustment system adapted to the Spanish healthcare environment that allows the general population to be stratified into morbidity groups. It has been constructed from qualitative–quantitative models where the care needs of users are collected according to variables such as mortality, risk of admission, Primary Care visits, or prescription. Its results can be used both in clinical-care management, as well as in epidemiology and healthcare administration. This system is based on the presence of chronic diseases, integrating all available medical diagnoses from the Primary Care medical history, and also considers recent acute diagnoses, through the Basic Minimum Hospital Data Set (BMHDS), as well as the date of diagnosis. It classifies the population into six morbidity groups, divided in turn into five levels of complexity, plus a healthy population group. Thus, the population is grouped into 31 mutually exclusive categories [18]. In addition, it provides additional results at an individual level, including the number of chronic diseases, the number of organ systems affected by a chronic disease, a clinical summary label, and the AMG weight of complexity or multi-morbidity index. The multi-morbidity index is calculated as the sum of the weights assigned to each of the pathology groups that affect the patient (groups that have an individual weight ranging from 0.14 for attention deficit disorder to 4.99 in the case of multiple myeloma). It is the population aggregator chosen by the Ministry of Health, Consumption and Social Welfare for the implementation of the Strategy for the Approach to Chronicity in the Spanish National Health System. The AMG system has been used in 13 Autonomous Communities, with a positive assessment of the results obtained [19].
The AMG system performs highly in accounting for relevant health outcomes and can be useful for clinical practice, resource planning and public health research [20]. Some of the studies carried out with this aggregator prove its concordance between the classification and the level of allocation by family doctors [21,22]; its capacity as a stratification tool in different chronic pathologies [23,24,25] and its value to refine the classification of more complex patients [26,27]. Furthermore, it has been compared with other stratifiers [27,28,29] proving to be, in general, a better predictor than these in relation to the use of certain healthcare resources (PC and Specialised Care (SC) consultations, emergencies, admissions). However, we found no studies that relate it to total costs.
Cost and stratifier studies such as the AMG focus their objectives on explaining cost variations, predicting consumption, building predictive models for budget allocation, obtaining the relative weight of costs, improving budget allocations, or analysing predictive capacity by adding other variables. They assume that the cost classifications in the stratification are adjusted to the expenditure per patient. In this study, we wanted to find out whether healthcare expenditure is similar for users with similar scores in the Adjusted Morbidity Groups classification system, or if on the contrary there is variability not explained by the AMG calculations. The objective was to analyse the variability of healthcare expenditure, within each of the groups generated by the stratifier and among different groups.

2. Materials and Methods

2.1. Design and Scope

An analytical and retrospective observational study based on data from health information system records. The study was carried out in the autonomous community of the Canary Islands (Spain) taking data from the years 2017 and 2018. Predictor variables were considered as of 31 December 2017, and total expenditure was calculated in 2018.

2.2. Participants

The study population comprised people over 15 years of age assigned to the health card database of the Canary Islands Health Service (SCS) throughout the year 2018 (1,691,075). The exclusion criteria were being under 15 years of age or being a mutual insurance society member (patients cared for by State mutual societies that provide social benefits and healthcare solely to their officials).
Of the study population, 51.04% (n = 863,071) were female and the mean age was 46.6 years (SD 17.9). A total of 17.39% (n = 294,032) of the population were over 65 years of age, and 23.21% (n = 392,520) were pensioners. The most frequent level of pharmaceutical contribution was 40%, with a total of 55.28% (n = 934,798).
In the analysis by sex it can be seen that 19% (n = 164,316) of females were over 65 years of age compared to 15.7% (n = 129,676) of males. Regarding the type of social security membership, 25.1% (n = 216,630) of females were pensioners compared to 21.2% (n = 175,536) of males. Regarding the percentage of pharmaceutical contribution, females have higher percentages than males in the 0% groups (n = 88.644 in females vs. n = 56,444 in males) and 10% (n = 187,175 vs. n = 160,439), and lower percentages in the rest.

2.3. Variables and Information Sources

The outcome variable was cost, calculated for the whole of 2018, of consultations and emergencies in public and state-subsidised private centres, hospitalisations, major outpatient surgery and prescriptions dispensed in pharmacies (Appendix A).
The sociodemographics, obtained on 31 December 2017, included sex, age, type of social security membership, percentage of pharmaceutical contribution; predictor variables related to the AMG adjustment system (complexity weight and AMG morbidity group); and clinical variables (Barthel index < 60, Pfeiffer questionnaire ≥ 5, having 2 or more hospital admissions in the last 12 months, and the number of active pharmaceutical ingredients) (Appendix A).
Different data sources were used to obtain the information depending on its nature. The variables sex, age, type of Social Security membership and type of pharmaceutical contribution were obtained from the health card database of the Canary Islands Health Service. The variables of medical consultations in the Primary Care centre, medical consultations at home, nursing consultations in the Primary Care centre, nursing consultations at home, and outpatient emergencies in public centres were obtained from the Primary Care Health History. The variables of hospital emergencies in public centres and specialised care medical consultations in public centres were collected from the Clinical History of Hospital Care. The variables of hospitalisations in public centres, home hospitalisations in public centres, episodes of outpatient major surgery in public centres, hospitalisations in state-subsidised private centres, episodes of major outpatient surgery in state-subsidised private centres were obtained from the Minimum Basic Data Set of the Specialised Care Register (CMBD-RAE). The variables of state-subsidised private emergencies, specialised care medical consultations in state-subsidised private centres, physiotherapy treatments in state-subsidised private centres and rehabilitation consultations in state-subsidised private centres were collected from the State-Assisted Hospitals Information System (SICH). Finally, the data on prescriptions dispensed in pharmacies charged to the Canary Islands Health Service, whether electronic or on paper, were obtained from the Electronic Prescription of the Canary Islands Health Service (REC-SCS).
The key unit of integration of all the variables from the different health information systems was the patient’s file number, which was encrypted and transformed into a single code for each individual.

2.4. Statistical Analysis

The statistical analysis was performed using the statistical program R Commander 2.6-2, graphical user interface of the statistical package R 4.0.0. The level of significance of p < 0.05 was adopted. An analysis consisting of descriptive statistics and bivariate analysis to obtain the relationship between explanatory and outcome variables was carried out, including an analysis of variance to study the effect of morbidity groups on costs, and a multiple linear regression to study the relationship between the weight of complexity and cost.
First, a descriptive analysis of all the global and group study variables, as well as by sex, was carried out to describe the characteristics of the study subjects. Secondly, a bivariate analysis was performed to study the relationship between each independent variable and the use of resources in the year immediately after. We used corresponding evidence to compare quantitative variables in two or more groups. In the case of the bivariate association study between qualitative variables, the Ji-square test or Fisher’s exact test was used. For the association between quantitative variables, the Student-Fisher t-test (two groups) or ANOVA (more than two groups) were used. In the event that the conditions for the application of the parametric tests were not met, the corresponding non-parametric tests were used.
Specifically, to calculate the influence of the AMG morbidity group on the patient’s total costs, we then conducted an analysis of the variability between groups and intra-groups by means of an analysis of variance (ANOVA), using the coefficient η2 [30] as follows:
  • The sum of the factor squares (between groups) measures the variability between factor levels.
  • The sum of the residual squares (intra-group) measures the variability within each level, that is, the variability that is not due to the qualitative variable or factor. It is calculated as the sum of the squares of the deviations of each observation with respect to the average of the level to which it belongs.
  • The sum of total squares measures the total variability of the data and is defined as the sum of the squares of the differences of each observation with respect to the general average of all observations.
  • To define the magnitude of the effect with this coefficient η2, we used a convention in which >0.01: small, >0.06: medium, >0.14: large.
Finally, to study the association between the explanatory variables and the outcome variables, a multiple linear regression model was constructed taking total costs as a dependent variable. The logarithm was applied to normalise the cost since it does not follow a normal distribution.

2.5. Ethical Considerations

The ethical principles of the Declaration of Helsinki and the standards of Good Clinical Practice in research were followed. The study was approved on 29 January 2021, by the clinical research ethics committee of the Dr. Negrín University Hospital of Gran Canaria (Code: 2021-037-1). The study complies with the provisions of Organic Law 3/2018, of 5 December, on the Protection of Personal Data and guarantee of digital rights, as well as Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016, on Data Protection.

3. Results

3.1. Descriptive Analysis

3.1.1. Clinical Variables

The distribution of the study population by morbidity, sex and age is shown in Table 1. As age increases, comorbidity increases: the increase in females compared to males in the group affected by four or more systems is notable. The complexity of our population, according to the AMG, has an average weight of 5.82 (SD 5.89) with quartiles of 1.63, 4.13 and 8.14.
A total of 1.35% (n = 22,800) of the population had two or more hospital discharges in 2017 and 0.37% (n = 6308) of the population had a Barthel index of less than 60 points (with 1.36% (n = 22,998) of the population assessed at least once), and 0.43% (n = 7276) had a Pfeiffer index greater than or equal to 5 points (with 3.37% (n = 56,989) of the population assessed). The mean number of drugs prescribed in the population was 4.35 (SD 5.1) and 36.92% (n = 624,344) were polymedicated patients (with 5 or more drugs).
Broken down by sexes, the mean (6.52) weight of AMG complexity was higher in females compared with males (5.10). A total of 0.5% (n = 4315) of females had a Barthel index of less than 60 points compared to 0.2% (n = 1656) of males. In the Pfeiffer questionnaire greater than or equal to five points, 0.6% (n = 5178) of females were observed compared to 0.2% (n = 1656) of males. A total of 1.4% (n = 12,082) of females had two or more incomes in 2017 compared to 1.3% (n = 10,764) of males. Females consumed an average of 5.19 active pharmaceutical ingredients compared to 3.48 for males. Patients with two or more hospital discharges were older than those who did not meet this criterion (61.5 vs. 46.4 years), the same occurred with those who had a Barthel index under 60 versus those who did not (79.9 vs. 46.5 years) and those who had a Pfeiffer index greater than or equal to 5 (80.0 vs. 46.5 years). In addition, age was positively correlated with AMG weight (Spearman’s rho = 0.47). Table 1 shows the mean ages of each morbidity group.

3.1.2. Average Costs

The cost analysis produced an average total cost per patient in 2018 of €1283.08 (SD 3007), with the highest item being Specialised Care (€571.90, SD 2274) followed by the dispensing of prescription drugs (€345.52, SD 946), Primary Care (€246.85, SD 421) and state-subsidised private centres (€118.80, SD 808). The average total cost for females was €1377.04 versus €1185.14 for males.

3.2. Bivariate Analysis

The bivariate analysis between the total cost of the year 2018 and the quantitative variables showed a significant correlation with the AMG weight (rho = 0.678) and the prescribed active ingredients (rho = 0.689), but not with age (rho = 0.386).
In relation to the categorical variables, the average total cost for females was €1377.04 (SD 2868.27; CI 95%: 1370.99–1383.09) vs. €1185.14 (SD 3142.05; CI 95%: 1178.37–1191.91) in males; €2950 (SD 2469.06; CI 95% 2934.28–2966.34) in those over 65 vs. €932 (SD 4435.34; CI 95% 928.09–936.28) in those under that age; €2880 (SD 4575.85; CI 95% 2865.98–2894.61) in pensioners vs. 800 (SD 2107.49; CI 95% 796.66–803.91) in active subjects; the differences are even greater between people with a Barthel index under 60 compared with the rest (€6058 (SD 6079.87; CI 95% 5908.14–6208.27) vs. €1265 (SD 2975.15; CI 95% 1260.71–1269.69)), people with a Pfeiffer index greater than or equal to 5 compared to the rest (€4877 (SD 4975.91; CI 95% 4762.35–4991.05) vs. €1268 (SD 2986.27; CI 95% 1263.04–1272.06)) and among people who have had two or more discharges in the previous year than those who have not (€5808 (SD 2818.28; CI 95% 5700.65–5915.84) vs. €1221 (SD 8288.82; CI 95% 1216.96–1225.51). Where there does not seem to be a linear relationship is between the percentage of pharmaceutical contribution and total costs (Figure 1), noting that a lower contribution reflects higher costs.
Regarding the relationship between morbidity groups and costs (Table 2), the ANOVA analysis found significant differences between the total mean expenditure of the different morbidity groups of the AMG system, although each of the morbidity groups had a high variability internally (coefficient of variation greater than 1). Otherwise, the result of the coefficient η2 in the ANOVA analysis between groups through the sum of the squares of the factor (1,393,131,734,896), divided by the sum of the total squares: (1,393,131,734,896 + 13,897,147,949,253 = 15,290,279,684,149) stood at 0.09, and the influence of the Morbidity group AMG on the total costs of the patient, therefore, had a magnitude of medium effect (Table 3).

3.3. Multivariant Analysis

In a first linear regression, the explanatory variables include: high in 2017 greater than or equal to two, Barthel index under 60, age, being female, being a pensioner, AMG weight, Pfeiffer equal to or greater than 5, percentage of pharmaceutical contribution and number of active ingredients prescribed were significantly related to the logarithm of the total cost, accounting for 36.43% (R2 = 0.3643; p < 0.001) of the model (Table 4). The AMG alone in relation to cost has an R2 = 0.31.
In a second model constructed with the number of active ingredients, AMG weight, being female and being a pensioner, which were the variables that most correlated with costs in the previous model, the variability of the total cost algorithm was explained by 36.4% (p < 0.001) (Table 5).

4. Discussion

The study population was younger than the Spanish average, with 19.58% of people over 65 years of age [31], with a low income since more than 50% had a level of pharmaceutical contribution of 40%, which is equivalent to an income of less than €18,000 per annum, and with higher morbidity than that described in the validation study of the AMG [18]. Furthermore, the average number of drugs prescribed in our population was 4.35, above that found by a similar study of 3.40 [32].
On the other hand, in the descriptive analysis it was observed that the female population presented more functional and cognitive impairment, a greater degree of complexity, consumed more active ingredients and had more hospital admissions than males, which is consistent with other studies [7]. These results and the underdiagnosis of disability [33], are aspects that would require a specific analysis.
However, there was a correlation between the total cost and the weight variables of AMG and the morbidity group, a finding that coincides with other studies where the average total healthcare expenditure increased with the number and intensity of chronic diseases [34]. This correlation also occurred with prescribed active ingredients and to a lesser extent with age, as seen in other studies [35], and was not significant with the rest of the variables. Another variable generated by the AMG is the weight of complexity which, in relation to costs, is only the second most important factor, the first being the number of prescribed active ingredients, which comes after other variables such as being female and a pensioner.
In this study, it was observed that the average cost assigned to each of the seven groups of morbidity is different, increasing with complexity, with the exception of the pregnancy and childbirth group. However, within each group, there was a certain degree of heterogeneity since the coefficient of variation was higher than one in all cases. All this was reflected in an η2 value that indicated a medium effect of this variable on the variance of the total cost, and we could therefore deduce that the morbidity group assigned to each patient was not sufficient to explain their costs. These results were difficult to compare since we could not review other studies that valued this metric of relationship with the total costs for any stratifier.
An adjustment system can be considered acceptable if it can explain between 20% and 25% of the variance between individuals in resource use [36]. In our case, the AMG system did not by itself achieve this explanatory power, but both the model created in our study and the reviewed studies shown below exceeded this value. Even so, it would be interesting to approximate it to the maximum of adjustment by adding other inputs given the scarcity of resources and the importance of equity for a healthier society. More complex (more comprehensive) models of risk adjustment would better predict cost and would have better performance [37].
The explanatory power of the four variables of our model on costs (female, pensioner, AMG weight and prescribed active ingredients) was 36.4%. Comparison with other studies is difficult because they use different stratifiers that measure morbidity and severity, but not with the same inputs or algorithms, and use different types of research and analysis. Although according to several authors, these stratifiers present similarities, we believe that they are only approximately comparable, but for the purpose of our research, we find it interesting to comment on them. For example, in a study where the CRG was used including a population of all ages, the model composed of state of health, severity, age and sex reached an R2 of 49.4% of total expenditure explanation [34]. Another study, which also analysed a sample of all age groups, found a predictive model consisting of age, sex, aggregate diagnostic groups (ADGs) and Rx-MGs (drugs) obtained an R2 of 38.2%, which increased to 46.5% when adding the total cost of the previous year [38]. In another study, conducted only in a primary care setting, the model that included age, sex and ACG obtained an explanatory power R2 of 42.6% (p < 0.001) of the total expenditure while the ACG classification explained the variance of this total cost of primary care by 51.6% [10]. In the Orueta study, where the analysis was carried out with the entire population aged over 14 years of the community, using two-stage models and hierarchical models, three stratifiers–ACG-PM, CRG and DCG-HCC–were compared, obtaining the best R2 with respect to the total costs, a model that adds the previous costs to the stratifier data. The results were 26.9% for DCG-HCC, 26% for ACG and 25.4% for CRG (which in this case did not include prescription information) [32]. Coderch’s work constructed a linear regression model with age, ACRG3 (part of the CRG classification), previous expenditure in pharmacies and expenditure on hospital medication, achieving an R2 of 38.2% [35].
The strengths of our study include the large sample size, the inclusion of the cost of primary care, and both public and private hospitals with state funding. Another strength is that we analysed data from more recent years than in the studies reviewed, which meant an improvement in the number of records and the maturity of the information systems. However, it is necessary to clarify some limitations in our study, such as the lack of a standardised method to calculate the cost per patient in our healthcare system, which meant we had to calculate them based on preset prices, accepting that there are some services that are not clearly identified as home hospitalisation, mental health or day hospital, inter alia. In addition, this study does not use the previous total costs, which, according to Chang [39], are a more powerful tool than stratification in terms of correctly identifying users with high expenses, stating that there is still much room for improvement in diagnostic-based models. Another aspect to consider is the low number of functional assessments (Barthel index) and cognitive assessments (Pfeiffer test) carried out since they are two indicators of quality of life that can provide a more accurate picture of the state of health of our population with multi-morbidity.
Another general limitation of stratifiers that needs to be considered is that the quality of the results is subject to the quality of the clinical history records. In a study with ACG to estimate costs in PC, the weighting of the ACG seems to be sensitive to the precision with which doctors include diagnoses in clinical histories [40]. This may mean that various populations and studies are not comparable. Depending on the sensitivity of health policies, pathologies are underestimated or overestimated by professionals who are more sensitised to the health programmes offered by their Health Services. It is also known that in primary care there is usually an under-reporting of diagnoses, although this disadvantage does not appear to be important enough to affect the results since a requirement of these adjustment models is to work with imperfect data in a real world [32].
The studies reviewed show that the inputs added to the stratifiers are insufficient to predict the cost beyond 50%, and often adding other factors to the stratifier such as certain clinical variables and previous costs [35,38]. Although the information on morbidity and co-morbidity is included, the heterogeneity of clinical cases makes future cost estimates difficult [36]. In fact, all these stratifiers stem only from medical diagnoses, age, sex, contacts with the healthcare system, surgical procedures or pharmaceutical prescription, and do not take account of determinants for health such as personal circumstances including education, purchasing power, lifestyles, social context, fragility, resilience to problems or nursing care. All of these factors may be causes of unexplained variability. If the central role of the aggregators is to define what complexity is, we need to add other inputs so that the algorithm refines its classification. Future research should be along these lines to improve these tools that help managers and healthcare professionals. The creators of the AMG themselves comment on the need to include other factors such as psychosocial problems, which would add greater value to the multifactorial equation [18]. As Starfield [41] states, the burden of morbidity interacts with social and economic factors, which presents a challenge to measure this complexity.
The AMG is a very recent stratification system, and future research will be able to approximate data. Specifically, our study proves that the variability of healthcare expenditure in users with similar AMG scores is significant. Future studies may improve the accuracy in classifying people through the inclusion of inputs that are not currently considered and that also determine the complexity, such as lifestyle habits, risk factors, complying with healthcare recommendations, socioeconomic conditions, place of residence, unwanted solitude, or nursing care. We know that nursing care has been shown to influence the prediction of total expenditure by improving the explanatory power of resource use from 13.70% (sociodemographic variables) to 19.77% when nursing diagnoses are factored in [42].

5. Conclusions

There is variability in expenditure among the seven groups classified by the AMG, which is consistent when classified by morbidity. However, there is also high variability in users with similar scores and classified in the same AMG group, which may indicate that the stratifier needs to be improved. The model created in the study explains 36.4% of the variability in total expenditure, matching the predictive ability of the adjustment models, which remains small to medium. Morbidity will be the marker of expenditure in public and private health care systems, so an improvement of the stratifiers should be taken into account.
There have already been some attempts such as the inclusion of socio-demographic variables or functional and cognitive assessment, readmissions and prescribed therapeutic groups. This room for improvement may contribute to the more equitable distribution of resources, greater satisfaction of professionals when they see that their work is more accurate, and a more precise tool for managers.

Author Contributions

Conceptualisation, M.C.C.-S.; methodology, M.C.C.-S., V.M.G.-C.; formal analysis, M.C.C.-S., M.I.O.-C.; investigation, M.C.C.-S., V.M.G.-C., M.I.O.-C.; data curation, M.C.C.-S., V.M.G.-C.; writing—original draft preparation, M.C.C.-S., V.M.G.-C., M.I.O.-C.; writing—review and editing, M.C.C.-S., V.M.G.-C., M.I.O.-C.; visualisation, M.C.C.-S.; supervision, V.M.G.-C., M.I.O.-C.; project administration, M.C.C.-S., V.M.G.-C., M.I.O.-C. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the clinical research ethics committee of Dr. Negrin University Hospital of Gran Canaria (Code: 2021-037-1).

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

We would like to thank Marcos Estupiñán Ramírez for his help in the statistical analysis.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

  • Glossary of variables
  • Cost variables:
  • Medical consultations in the primary care centre (€28.78/consultation) [COSTE_AP_MED_CEN_18]
  • Home primary care medical consultations (€43.66/consultation) [COSTE_AP_MED_DOM_18]
  • Nursing consultations in the Primary Care centre (€26.62/consultation) [COSTE_AP_ENF_CEN_18]
  • Home primary care nursing consultations (€106.47/consultation) [COSTE_AP_ENF_DOM_18]
  • Primary care midwifery consultations (€59.32/consultation) [COSTE_AP_MAT_18]
  • Social worker consultations in primary care (€176.24/consultation) [COSTE_AP_TS_18]
  • Outpatient medical emergencies (PC) in public centres (€28.78/consultation) [COSTE_AP_URG_MED_18]
  • Outpatient nursing emergencies (PC) in public centres (€26.62/consultation) [COSTE_AP_URG_ENF_18]
  • Hospital emergencies in public centres (€227.78/episode) [COSTE_AE_URG_18]
  • Specialised care medical consultations as a first visit in public centres (€137.22/consultation) [COSTE_AE_CONS_PRIMERAS_18]
  • Successive specialised care medical consultations in public centres (€82.33/consultation) [COSTE_AE_CONS_SUCESIVAS_18]
  • Emergencies in state-subsidised private centres [COSTE_CC_URG_18]
  • Specialised care medical consultations in state-subsidised private centres [COSTE_CC_CONS_18]
  • Physiotherapy treatments in state-subsidised private centres [COSTE_CC_FISIO_18]
  • Rehabilitation consultations in state-subsidised private centres [COSTE_CC_REHAB_18]
  • Hospitalisations and outpatient major surgery. In these cases, a standard cost per episode is used, based on the applicable related diagnostic group. (GDR-AP):
  • Hospitalisations in public centres [COSTE_AE_HOSP_18]
  • Home hospitalisations in public centres (HADO) [COSTE_AE_HADO_18]
  • Major outpatient surgery in public centres [COSTE_AE_CMA_18]
  • Hospitalisations in private centres (including private and state-subsidised private activity) [COSTE_CC_HOSP_18]
  • Major outpatient surgery in private centres (including private and state-subsidised private activity) [COSTE_CC_CMA_18]
  • Prescriptions dispensed in pharmacies of the Canary Islands Health Service, whether electronic or on paper. [COSTE_REC_DISP_18]
  • Sociodemographic:
  • FEMALE: Sex of the patient (1: Female, 0: male)
  • AGE: Age in years at the start of the study period
  • PENSIONER: Type of Social Security membership
  • PERCENTAGE_CONTRIBUTION_PHARMA: Percentage of pharmaceutical contribution: 0, 10, 40, 50 or 60%
  • WEIGHT_AMG: Weight of AMG complexity, number obtained based on the sum of the weights associated with each group of diagnoses in which the patient is included.

References

  1. Goncalves, P.; López, M.; Gil, J. Informe Cronos: Hacia el Cambio de Paradigma en la Atención a Enfermos Crónicos. The Boston Consulting Group. 2014. Available online: https://docplayer.es/2163643-Informe-cronos-hacia-el-cambio-de-paradigma-en-la-atencion-a-enfermos-cronicos-informe-elaborado-por-the-boston-consulting-group.html (accessed on 9 August 2021).
  2. World Health Organization. Addressing Comorbidity between Mental Disorders and Major Noncommunicable Diseases. 2017. Available online: https://www.euro.who.int/en/publications/abstracts/addressing-comorbidity-between-mental-disorders-and-major-noncommunicable-diseases-2017 (accessed on 15 August 2021).
  3. Fernández-Niño, J.A.; Bustos-Vázquez, E. Multimorbilidad: Bases conceptuales, modelos epidemiológicos y retos de su medición. Biomédica 2016, 36, 188. [Google Scholar] [CrossRef]
  4. Inoriza, J.M.; Coderch, J.; Carreras, M.; Vall-Llosera, L.; García-Goñi, M.; Lisbona, J.M.; Ibern, P. Measurement of morbidity attended in an integrated health care organization. Gac. Sanit. 2009, 23, 29–37. [Google Scholar] [CrossRef] [Green Version]
  5. Bähler, C.; Schoepfer, A.M.; Vavricka, S.R.; Brüngger, B.; Reich, O. Chronic comorbidities associated with inflammatory bowel disease: Prevalence and impact on healthcare costs in Switzerland. Eur. J. Gastroenterol. Hepatol. 2017, 29, 916–925. [Google Scholar] [CrossRef]
  6. Barceló, A.; Epping-Jordan, J.; Orduñez, P.; Luciani, S.; Agurto, I.; Tasca, R. Cuidados Innovadores para las Condiciones Crónicas: Organización y Prestación de Atención de Alta Calidad a las Enfermedades Crónicas no Transmisibles en las Américas. Pan American Health Organization. 2013. Available online: https://www.paho.org/hq/dmdocuments/2013/CuidadosInnovadores-v5.pdf (accessed on 15 August 2021).
  7. Ministerio de Sanidad Servicios Sociales e Igualdad. Estrategia para el Abordaje de la Cronicidad en el Sistema Nacional de Salud Estrategia Aprobada por el Consejo Interterritorial del Sistema Nacional de Salud el 27 de Junio de 2012. Available online: https://www.sanidad.gob.es/organizacion/sns/planCalidadSNS/pdf/ESTRATEGIA_ABORDAJE_CRONICIDAD.pdf (accessed on 25 March 2022).
  8. Cabo, J.; Bellemont, M.A.; Cabo, J.; Cabo, V. Ajustes de riesgos y calidad asistencial. Agrupadores (APG, GRD, AP-GRD, IR-GRD, ACG, DxCG, CRG). In Gestión de la Calidad en las Organizaciones Sanitarias; Edición Díaz de Santos: Madrid, Spain, 2014; pp. 1154–1205. [Google Scholar]
  9. Huang, X.; Peterson, S.; Lavergne, R.; Ahuja, M.; McGrail, K. Predicting the Cost of Health Care Services: A Comparison of Case-mix Systems and Comorbidity Indices That Use Administrative Data. Med. Care 2020, 58, 114–119. [Google Scholar] [CrossRef] [PubMed]
  10. Sicras-Mainar, A.; Velasco-Velasco, S.; Navarro-Artieda, R.; Jodar, A.A.; Plana-Ripoll, O.; Hermosilla-Pérez, E.; Bolibar-Ribas, B.; Prados-Torres, A.; Violan-Fors, C. Obtaining the mean relative weights of the cost of care in Catalonia (Spain): Retrospective application of the adjusted clinical groups case-mix system in primary health care. J. Eval. Clin. Pract. 2013, 19, 267–276. [Google Scholar] [CrossRef] [PubMed]
  11. Tarazona, V.C.; Olmeda, N.G.; Consuelo, D.V.; Collado, A.C. Impact of Morbidity on Health Care Costs of a Department of Health through Clinical Risk Groups. Valencian Community, Spain. Rev. Esp. Salud Publica 2016, 90, e1–e15. [Google Scholar]
  12. Vivas-Consuelo, D.; Usó-Talamantes, R.; Trillo-Mata, J.L.; Caballer-Tarazona, M.; Barrachina-Martínez, I.; Buigues-Pastor, L. Predictability of pharmaceutical spending in primary health services using Clinical Risk Groups. Health Policy 2014, 116, 188–195. [Google Scholar] [CrossRef] [PubMed]
  13. Sicras-Mainar, A.; Serrat-Tarres, J.; Navarro-Artieda, R.; Llausí-Sellés, R.; Ruano-Ruano, I.; González-Ares, J.A. Adjusted Clinical Groups use as a measure of the referrals efficiency from primary care to specialized in Spain. Eur. J. Public Health 2007, 17, 657–663. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. De Miguel, P.; Caballero, I.; Rivas, F.J.; Manera, J.; de Vicente, M.A.; Gómez, Á. Morbilidad observada en un área sanitaria: Variabilidad e impacto en profesionales y en financiación. Aten Primaria 2015, 47, 301–307. [Google Scholar] [CrossRef] [Green Version]
  15. Alberquilla, A.; Feliz, L.V.; de Francisco, A.G.; Vásquez, A.R.; del Mar Rodríguez, M.; de Francisco, S.G. Análisis de la hiperfrecuentación ajustando por comorbilidad según ACG (Adjusted Clinical Groups) en una Zona Básica de Salud. Med. Gen. 2017, 6, 246–251. [Google Scholar] [CrossRef]
  16. Aguado, A.; Guinó, E.; Mukherjee, B.; Sicras, A.; Serrat, J.; Acedo, M.; Ferro, J.J.; Moreno, V. Variability in prescription drug expenditures explained by adjusted clinical groups (ACG) case-mix: A cross-sectional study of patient electronic records in primary care. BMC Health Serv. Res. 2008, 8, 1–11. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. García-Goñi, M.; Ibern, P. Predictability of drug expenditures: An application using morbidity data. Health Econ. 2008, 17, 119–126. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Monterde, D.; Vela, E.; Clèries, M. Adjusted morbidity groups: A new multiple morbidity measurement of use in Primary Care. Aten. Primaria 2016, 48, 674–682. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  19. Ministerio de Sanidad Consumo y Bienestar Social. Informe del Proyecto de Estratificación de la Población por Grupos de Morbilidad Ajustados (GMA) en el Sistema Nacional de Salud (2014–2016). Available online: https://www.sanidad.gob.es/ca///organizacion/sns/planCalidadSNS/pdf/informeEstratificacionGMASNS_2014-2016.pdf (accessed on 25 March 2022).
  20. Vela, E.; Clèries, M.; Monterde, D.; Coca, M.; Valero-Bover, D.; Piera-Jiménez, J.; Eroles, L.G.; Sust, P.P. Performance of Quantitative Measures of Multimorbidity: A Population-Based Retrospective Analysis. BMC Public Health 2021, 21, 1881. [Google Scholar] [CrossRef]
  21. González, A.I.; Miquel, A.M.; Rodríguez, D.; Pascualc, M.H.; Perrucad, L.S.; Herrerae, I.M.; Grupo de Trabajo GMA y Grupo de Estratificación del Servicio Madrileño de Salud. Concordancia y utilidad de un sistema de estratificación para la toma de decisiones clínicas. Aten. Primaria 2017, 49, 240–247. [Google Scholar] [CrossRef] [Green Version]
  22. Clèries, M.; Monterde, D.; Vela, E.; Guarga, À.; García, L.E.; Pérez, P.S. Clinical validation of 2 morbidity groups in the primary care setting. Aten. Primaria 2020, 52, 96–103. [Google Scholar] [CrossRef]
  23. Cancio, J.M.; Vela, E.; Santaeugènia, S.; Clèries, M.; Inzitari, M.; Ruiz, D. Influence of demographic and clinical characteristics of elderly patients with a hip fracture on mortality: A retrospective, total cohort study in North-East Spain. Bone 2018, 117, 123–129. [Google Scholar] [CrossRef]
  24. Amblàs-Novellas, J.; Santaeugènia, S.J.; Vela, E.; Clèries, M.; Contel, J.C. What lies beneath: A retrospective, population-based cohort study investigating clinical and resource-use characteristics of institutionalized older people in Catalonia. BMC Geriatr. 2020, 20, 187. [Google Scholar] [CrossRef]
  25. Cortes, J.B.; Fernández, C.S.; de Oliveira, M.B.; Lagos, C.M.; Martínez MT, B.; Hernández, C.L.; del Cura González, I. Enfermedades crónicas en población pediátrica: Comorbilidades y uso de servicios en atención primaria. Pediatría 2020, 93, 183–193. [Google Scholar] [CrossRef]
  26. Bretos-Azcona, P.E.; Sánchez-Iriso, E.; Cabasés, J.M. Tailoring integrated care services for high-risk patients with multiple chronic conditions: A risk stratification approach using cluster analysis. BMC Health Serv. Res. 2020, 20, 806. [Google Scholar] [CrossRef]
  27. Estupinan-Ramirez, M.; Tristancho-Ajamil, R.; Sánchez-Janáriz, H. Comparison of predictive models for the selection of high-complexity patients. Gac. Sanit. 2019, 33, 60–65. [Google Scholar] [CrossRef] [PubMed]
  28. Monterde, D.; Vela, E.; Clèries, M.; Eroles, G.; Sust, P. Validity of adjusted morbidity groups with respect to clinical risk groups in the field of primary care. Aten. Primaria 2019, 51, 153–161. [Google Scholar] [CrossRef] [PubMed]
  29. Arias-López, C.; MP, R.V.; Sánchez, S. Validity of predictive power of the Adjusted Morbidity Groups (AMG) with respect to others population stratification tools. Rev. Esp. Salud Publica 2020, 94, 1–9. Available online: https://www.mscbs.gob.es/biblioPublic/publicaciones/recursos_propios/resp/revista_cdrom/VOL94/ORIGINALES/RS94C_202007079.pdf (accessed on 25 March 2022).
  30. Jiménez, J.M.; Herrera, A.S.; Rojas, W.S. Fuentes de varianza e índices de varianza explicada en las ciencias del movimiento humano. Pensar Mov. Rev. Cienc. Ejerc. Y Salud 2002, 2, 70–74. [Google Scholar] [CrossRef] [Green Version]
  31. Instituto Nacional de Estadística. Proporción de Personas Mayores de Cierta Edad por Provincia. Instituto Nacional de Estadística. 2021. Available online: https://www.ine.es/jaxiT3/Tabla.htm?t=1488 (accessed on 25 March 2022).
  32. Orueta, J.F.; Nuño-Solinis, R.; Mateos, M.; Vergara, I.; Grandes, G.; Esnaola, S. Predictive risk modelling in the Spanish population: A cross-sectional study. BMC Health Serv. Res. 2013, 13, 269. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Observatorio Estatal de la Discapacidad. Mujer con Discapacidad y Enfermedades Crónicas: Género, Discapacidad y Cronicidad Informe de Resultados. 2019. Available online: www.observatoriodeladiscapacidad.info (accessed on 25 March 2022).
  34. Caballer-Tarazona, V.; Guadalajara-Olmeda, N.; Vivas-Consuelo, D. Predicting healthcare expenditure by multimorbidity groups. Health Policy 2019, 123, 427–434. [Google Scholar] [CrossRef] [PubMed]
  35. Coderch, J.; Sánchez-Pérez, I.; Ibern, P.; Carreras, M.; Pérez-Berruezo, X.; Inoriza, J.M. Predicción del riesgo individual de alto coste sanitario para la identificación de pacientes crónicos complejos. Gac. Sanit. 2014, 28, 292–300. [Google Scholar] [CrossRef] [Green Version]
  36. Buglioli Bonilla, M.; Ortún Rubio, V. Sistemas de ajuste por riesgo. Rev. Méd. Urug. 2000, 16, 123–132. Available online: https://www.rmu.org.uy/revista/2000v2/art4.pdf (accessed on 25 March 2022).
  37. Chang, H.Y. The impact of morbidity trajectories on identifying high-cost cases: Using Taiwan’s National Health Insurance as an example. J. Public Health 2014, 36, 300–307. [Google Scholar] [CrossRef] [Green Version]
  38. Kuo, R.N.; Lai, M.S. Comparison of Rx-defined morbidity groups and diagnosis- based risk adjusters for predicting healthcare costs in Taiwan. BMC Health Serv. Res. 2010, 10, 126. [Google Scholar] [CrossRef] [Green Version]
  39. Chang, H.Y.; Lee, W.C.; Weiner, J.P. Comparison of alternative risk adjustment measures for predictive modeling: High risk patient case finding using Taiwan’s National Health Insurance claims. BMC Health Serv. Res. 2010, 10, 343. [Google Scholar] [CrossRef] [Green Version]
  40. Engström, S.G.; Carlsson, L.; Östgren, C.J.; Nilsson, G.H.; Borgquist, L.A. The importance of comorbidity in analysing patient costs in Swedish primary care. BMC Public Health 2006, 6, 36. Available online: http://www.biomedcentral.com/1471-2458/6/36 (accessed on 25 March 2022). [CrossRef] [PubMed] [Green Version]
  41. Valderas, J.M.; Starfield, B.; Sibbald, B.; Salisbury, C.; Roland, M. Defining Comorbidity: Implications for Understanding Health and Health Services. Ann. Fam. Med. 2009, 7, 357–363. [Google Scholar] [CrossRef] [PubMed]
  42. Company-Sancho, M.C.; Estupiñán-Ramírez, M.; Sánchez-Janáriz, H.; Tristancho-Ajamil, R. The connection between nursing diagnosis and the use of healthcare resources. Enferm. Clin. 2017, 27, 214–221. [Google Scholar] [CrossRef] [PubMed]
Ijerph 19 04219 g001 550
Figure 1. Relationship between pharmaceutical contribution and total costs in euros.
Figure 1. Relationship between pharmaceutical contribution and total costs in euros.
Ijerph 19 04219 g001
Table
Table 1. Distribution of the population by morbidity group, sex and average age.
Table 1. Distribution of the population by morbidity group, sex and average age.
Total PopulationMalesFemalesAverage Age (SD)
Morbidity Group% (n)
Healthy population10.28 (173,846)13.2 (22,948)7.5 (13,038)41.3 (14.9)
Acute illness6.43 (108,798)8.3 (9030)4.7 (5113)36.0 (12.3)
Pregnancy or childbirth1.81 (30,591)0.03.5 (1071)32.6 (6.6)
Chronic illness in 1 system17.18 (290,462)20.5 (59,545)14.0 (40,665)39.4 (14.0)
Chronic illness in 2 or 3 systems32.12 (543,237)33.1 (179,811)31.2 (169,490)44.5 (16.6)
Chronic illness in 4+ systems30.70 (519,212)23.5 (122,015)37.6 (195,224)56.9 (18.1)
Active neoplasia1.47 (24,929)1.4 (349)1.5 (374)64.1 (14.5)
Table
Table 2. Average costs by morbidity group in euros.
Table 2. Average costs by morbidity group in euros.
Morbidity GroupAverage Cost
in Euros		Standard
Deviation		Coefficient of
Variation
Healthy population1691.0876.45
Acute illness4251.4823.48
Pregnancy or childbirth17182.5271.47
Chronic illness in 1 system5171.6323.16
Chronic illness in 2 or 3 systems9812.4362.48
Chronic illness in 4+ systems24194.0021.65
Active neoplasia41266.1481.49
Table
Table 3. Calculation of η2 values in the analysis of variance (ANOVA).
Table 3. Calculation of η2 values in the analysis of variance (ANOVA).
DfSum SqMean SqF ValuePr (>F)
GM_C *61,393,131,734,896232,188,622,483<2 × 10−16 ***
Residuals1,691,06813,897,147,949,2538,217,971
* GM_C, Variable Adjusted Morbidity Groups; *** Raised to 16.
Table
Table 4. Relation between total cost and independent variables.
Table 4. Relation between total cost and independent variables.
CoefficientsStandard ErrorT Valuep
Constant3.4850.007439.162<0.001
More than 2 discharges in 2017−0.1610.015−10.754<0.001
Barthel index less than 60 points−0.4240.029−14.488<0.001
Age−0.002<0.001−21.125<0.001
Female0.2530.00373.254<0.001
Pensioner0.2760.00641.026<0.001
AMG weight *0.1330.0004281.994<0.001
Pfeiffer index over 5 points−0.0390.027−1.447<0.001
Percentage of pharmaceutical contribution0.0005<0.0013.585<0.001
Amount of prescribed active ingredients0.183<0.001341.926<0.001
* AMG: Adjusted Morbidity Groups; Residual standard error: 2.211 on 1691066 degrees of freedom Multiple R-squared: 0.3643, Adjusted R-squared: 0.3643. p-value: <0.001.
Table
Table 5. Relationship between costs and most significant explanatory variables.
Table 5. Relationship between costs and most significant explanatory variables.
CoefficientsEstimateStd. ErrorT ValuePr (>|t|)
(Intercept)3.41013910.00285121196.02<0.001
Female0.25537050.003454373.93<0.001
Pensioner0.20735150.004779343.38<0.001
Weight of the AMG *0.13108630.0004630283.15<0.001
Active ingredients prescribed0.18188070.0005296343.44<0.001
* AMG: Adjusted Morbidity Groups; Residual standard error: 2.212 on 1691071 degrees of freedom. Multiple R-squared: 0.364, Adjusted R-squared: 0.364. p-value: <0.001.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Company-Sancho, M.C.; González-Chordá, V.M.; Orts-Cortés, M.I. Variability in Healthcare Expenditure According to the Stratification of Adjusted Morbidity Groups in the Canary Islands (Spain). Int. J. Environ. Res. Public Health 2022, 19, 4219. https://doi.org/10.3390/ijerph19074219

AMA Style

Company-Sancho MC, González-Chordá VM, Orts-Cortés MI. Variability in Healthcare Expenditure According to the Stratification of Adjusted Morbidity Groups in the Canary Islands (Spain). International Journal of Environmental Research and Public Health. 2022; 19(7):4219. https://doi.org/10.3390/ijerph19074219

Chicago/Turabian Style

Company-Sancho, Maria Consuelo, Víctor M. González-Chordá, and María Isabel Orts-Cortés. 2022. "Variability in Healthcare Expenditure According to the Stratification of Adjusted Morbidity Groups in the Canary Islands (Spain)" International Journal of Environmental Research and Public Health 19, no. 7: 4219. https://doi.org/10.3390/ijerph19074219

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop