Analysis on Effectiveness of Impact Based Heatwave Warning Considering Severity and Likelihood of Health Impacts in Seoul, Korea

Many countries are operating a heatwave warning system (HWWS) to mitigate the impact of heatwaves on human health. The level of heatwave warning is normally determined by using the threshold temperature of heat-related morbidity or mortality. However, morbidity and mortality threshold temperatures have not been used together to account for the severity of health impacts. In this study, we developed a heatwave warning system with two different warning levels: Level-1 and Level-2, by analyzing the severity and likelihood of heat-related morbidity and mortality using the generalized additive model. The study particularly focuses on the cases in Seoul, South Korea, between 2011 and 2018. The study found that the threshold temperature for heat-related morbidity and mortality are 30 °C and 33 °C, respectively. Approximately 73.1% of heat-related patients visited hospitals when temperature was between 30 °C and 33 °C. We validated the developed HWWS by using both the threshold temperatures of morbidity and mortality. The area under curves (AUCs) of the proposed model were 0.74 and 0.86 at Level-1 and Level-2, respectively. On the other hand, the AUCs of the model using only the mortality threshold were 0.60 and 0.86 at Level-1 and Level-2, respectively. The AUCs of the model using only the morbidity threshold were 0.73 and 0.78 at Level-1 and Level-2, respectively. The results suggest that the updated HWWS can help to reduce the impact of heatwaves, particularly on vulnerable groups, by providing the customized information. This also indicates that the HWWS could effectively mitigate the risk of morbidity and mortality.


Introduction
South Korea, Japan, China, Australia, and many countries in EU have experienced record-breaking heatwaves, especially in 2018 and 2019 [1][2][3][4][5]. It is expected that the intensity, duration, and frequency of heatwaves will increase globally due to climate change [6]. Additionally, the length of the longest heatwave event is projected to persist for more than a week and possibly up to a month in the late 21st century over the Korean Peninsula [7].
Since the extreme heat event is expected to affect human health [8][9][10][11][12], many countries have developed the heatwave warning systems (HWWSs) to reduce its impact [13][14][15][16][17][18]. A few studies have shown that the HWWSs have contributed to the reduction of excess mortality. In Philadelphia, USA, it was reported that the HWWS could have reduced the deaths of about 117 people from heatwaves during the period 1995-1998 [19]. In France, after the 2003 European heatwave, the Heat Health Watch Warning System was established. In the 2006 heatwave, 2065 deaths were seen, which was smaller than the 6452 deaths predicted based on an epidemiological model for 1975-2003 [20]. The Korea Meteorological Administration (KMA) has also developed its own Heat Watch Warning System, which has been operating since 2008 [9].
The HWWS should be able to predict the impact of heatwaves on human health from mild to severe and provide countermeasures. The severity of the health impact of heat-

Research Scope
The study area was Seoul, South Korea where the threshold temperature used in the Heat Watch Warning System was developed. The morbidity, mortality, and weather data for the period 2011-2018 summer (June to August) were used. During the same period, the daily maximum temperature in Seoul was 29.1 °C on average. The hottest year was 2018, and the average daily maximum temperature was 30.5 °C.
The study procedure is shown in Figure 1. First, heat-related morbidity and excess mortality were analyzed as proxies of health impacts from heatwaves. Next, a threshold temperature for heat-related morbidity and a threshold temperature for mortality were analyzed. Finally, the association between threshold temperatures and health impact proxies were analyzed in three ways. (1) The cumulative number of HRI patients with increasing temperature was compared to the two threshold temperatures. (2) The daily excess mortality was also qualitatively compared to the two threshold temperatures. (3) The performance of warning based on threshold temperature was evaluated using the model skill score.

Materials
Population of heat-related morbidity was obtained from Healthcare Bigdata of National Health Insurance Sharing Service of Korea. Mortality data were obtained from Statistics Korea's data on causes of deaths. All causes (excluding external causes) of mortality were used in this study; the ICD-10 (International Statistical Classification of Diseases-10th Revision) codes are from A to R. These data cover the period of 2011-2018.

Materials
Population of heat-related morbidity was obtained from Healthcare Bigdata of National Health Insurance Sharing Service of Korea. Mortality data were obtained from Statistics Korea's data on causes of deaths. All causes (excluding external causes) of mortality were used in this study; the ICD-10 (International Statistical Classification of Diseases-10th Revision) codes are from A to R. These data cover the period of 2011-2018.
HRI monitoring data obtained from the Korea Disease Control and Prevention were used as verification data to evaluate the performance of threshold temperature. Since Healthcare Bigdata includes all patients who visited the hospital, patients who were not affected by the heatwave are included as noise. On the other hand, the monitoring data are patients who were determined to have been affected by heatwaves among emergency room visitors by emergency medicine doctors. Therefore, we used this data as verification data in the performance analysis. All-cause mortality or death from certain diseases (e.g., 4 of 13 cardiovascular, respiratory, and cancer) have been adopted in heatwave studies [41][42][43][44][45][46]. This study adopted all-cause mortality in the study.
Weather data was obtained by downscaling to a 1 km spatial resolution the daily maximum temperature (Tmax), daily minimum temperature (Tmin), daily average relative humidity, and daily mean windspeed from the digital forecasts provided by the Korea Meteorological Administration (KMA). The Gaussian process regression model was used for downscaling [47]. Durations of heatwaves, tropical nights, and dry days were also used in this study. The heatwave was defined as the 90th percentile of Tmax during the period ranging from 2011 to 2018. The tropical night had a Tmin of 25 • C or higher. Days with precipitation below 1 mm/day were regarded as dry days.

Statistical Methods
A generalized additive model (GAM) was employed as a statistical method to quantitatively analyze the relationship between temperature and health. GAMs have been widely used in time-series research because they can express the influence of nonlinear confounding variables in a nonlinear manner using nonparametric functions [36,48]. The statistic model used in this study was expressed as follows: where E(Y) denotes the expected daily death counts in the mortality cases and patient counts in the morbidity cases; s(T max ) is the daily maximum temperature; s(H avg ) is the daily average humidity; wind is the daily average wind speed; hw is duration of heatwaves; tn is duration of tropical nights; dry is the duration of dry days; s(DOY) is the day of year; and s(sn) is the serial number of the date. The s denotes smooth, nonparametric functions. f(W) and f(yr) represent the day of the week and year, respectively, and were used as dummy variables. The day of the week was classified to three types (holidays, the day after the holiday, and other days). R 3.4.0 and the mgcv package were used for the GAM analysis. The threshold temperature was determined from smoothed curves of the GAM. Daily excess mortality was estimated as the difference between daily deaths and expected deaths. The expected daily deaths were calculated using the following Equation (2). This formula is based on the formula proposed by Jeong et al. [49] and applies the method of FluMOMO v4. 2 [50], which considers the population structure.
where E(y,d) is the number of expected deaths on the d-th day of year y, and E(y,d)ij is the number of expected deaths by gender (i) and age (j) on a given date. Age groups were divided at 5-year intervals, and those over 65 were categorized as one group. Mij is the average number of deaths per day for each gender and age group over the entire period. Wij(d) is the value obtained by dividing the number of deaths on the d-th day by Mij. When calculating Wij(d), a 7-day weighted moving average was applied to remove weekly variability. Weights 1, 2, 3, 4, 3, 2, and 1 were applied in 7 days. The highest weight is assigned to the days to be calculated. Wij(y, w) is the number of deaths on the w (day of the week) divided by the average number of deaths on the w in the y year. Wij(y) is the total number of deaths in year y divided by the annual average number of deaths in the entire period.

Evaluation Methods Using Skill Score
The area under the curve (AUC) was used as the skill score for evaluation. The AUC was calculated from the receiver operating characteristic (ROC) curve, drawn using the sensitivity (true positives/(true positives + false negatives)) and the specificity ((true negatives/(false positives + true negatives)) derived from the confusion matrix [51]. The R package pROC was used to calculate AUC [52].
The predicted event in the confusion matrix was the individual warning level, and the actual event was defined as a gradual increase in the number of patients or the occurrence of mortality. The warning to be evaluated consisted of two levels (Level-1 and Level-2). Level 1 is a period in which warning regarding an increase in the HRI morbidity is required, and level-2 is a period in which a warning regarding an increase in mortality is required.

Threshold Temperatures and HRI Patients
The effects of maximum temperature on heat-related morbidity and mortality showed a pattern as shown in Figure 2. The heat-related morbidity had a pattern of increasing rapidly from a relatively lower temperature than mortality. The threshold temperature for heat-related morbidity was 30 • C, while the threshold for mortality was approximately 33 • C.
total number of deaths in year y divided by the annual average number of deaths in the entire period.

Evaluation Methods Using Skill Score
The area under the curve (AUC) was used as the skill score for evaluation. The AUC was calculated from the receiver operating characteristic (ROC) curve, drawn using the sensitivity (true positives/(true positives + false negatives)) and the specificity ((true negatives/(false positives + true negatives)) derived from the confusion matrix [51]. The R package pROC was used to calculate AUC [52].
The predicted event in the confusion matrix was the individual warning level, and the actual event was defined as a gradual increase in the number of patients or the occurrence of mortality. The warning to be evaluated consisted of two levels (Level-1 and Level-2). Level 1 is a period in which warning regarding an increase in the HRI morbidity is required, and level-2 is a period in which a warning regarding an increase in mortality is required.

Threshold Temperatures and HRI Patients
The effects of maximum temperature on heat-related morbidity and mortality showed a pattern as shown in Figure 2. The heat-related morbidity had a pattern of increasing rapidly from a relatively lower temperature than mortality. The threshold temperature for heat-related morbidity was 30 °C, while the threshold for mortality was approximately 33 °C. The comparison between the cumulative number of heat-related morbidity with respect to the two threshold temperatures shows that most of the heat-related morbidity already occurred before the temperature reached 33 °C (Figure 3), which is the threshold temperature for mortality. Even if the cumulative number of heat-related morbidities was only counted above 30 °C, 53.5% of heat-related morbidity occurred before 33 °C ( Figure  4). The comparison between the cumulative number of heat-related morbidity with respect to the two threshold temperatures shows that most of the heat-related morbidity already occurred before the temperature reached 33 • C (Figure 3), which is the threshold temperature for mortality. Even if the cumulative number of heat-related morbidities was only counted above 30 • C, 53.5% of heat-related morbidity occurred before 33 • C ( Figure 4).
England, France, and the US issue warnings when the threshold temperature will be exceeded within the next few days [14,15,24,26,27]. This method can also be considered in Korea. Currently, Korea issues a warning when the mortality threshold temperature has been exceeded for two consecutive days. Therefore, there is a need for a way to issue a warning about an increase in heat-related illness before reaching the mortality threshold temperature. If a lower warning level is issued a few days before reaching the threshold temperature, is it possible to cover the heat-related morbidity that occurs before the threshold temperature of mortality is reached? viron. Res. Public Health 2021, 18, x FOR PEER REVIEW 6 of 13  England, France, and the US issue warnings when the threshold temperature will be exceeded within the next few days [14,15,24,26,27]. This method can also be considered in Korea. Currently, Korea issues a warning when the mortality threshold temperature has been exceeded for two consecutive days. Therefore, there is a need for a way to issue a warning about an increase in heat-related illness before reaching the mortality threshold temperature. If a lower warning level is issued a few days before reaching the threshold temperature, is it possible to cover the heat-related morbidity that occurs before the threshold temperature of mortality is reached? Table 1 shows that this is difficult. The number of days when the daily maximum temperature exceeded 30 °C and was below 33 °C was 194 during 2011 to 2018, in Seoul. If a warning was issued 2 days before the temperature reached 33 °C, the total number of days for which the warning was issued was 54. However, these days include 22 instances when the temperature did not exceed 30 °C. Therefore, the number of days for which warnings were issued when the temperatures ranged between 30 °C and 33 °C was only 32. Approximately 84% of the days when the daily maximum temperature was between the morbidity threshold and the mortality threshold were missed. As a result, it is highly  England, France, and the US issue warnings when the threshold temperature will be exceeded within the next few days [14,15,24,26,27]. This method can also be considered in Korea. Currently, Korea issues a warning when the mortality threshold temperature has been exceeded for two consecutive days. Therefore, there is a need for a way to issue a warning about an increase in heat-related illness before reaching the mortality threshold temperature. If a lower warning level is issued a few days before reaching the threshold temperature, is it possible to cover the heat-related morbidity that occurs before the threshold temperature of mortality is reached? Table 1 shows that this is difficult. The number of days when the daily maximum temperature exceeded 30 °C and was below 33 °C was 194 during 2011 to 2018, in Seoul. If a warning was issued 2 days before the temperature reached 33 °C, the total number of days for which the warning was issued was 54. However, these days include 22 instances when the temperature did not exceed 30 °C. Therefore, the number of days for which warnings were issued when the temperatures ranged between 30 °C and 33 °C was only 32. Approximately 84% of the days when the daily maximum temperature was between the morbidity threshold and the mortality threshold were missed. As a result, it is highly  Table 1 shows that this is difficult. The number of days when the daily maximum temperature exceeded 30 • C and was below 33 • C was 194 during 2011 to 2018, in Seoul. If a warning was issued 2 days before the temperature reached 33 • C, the total number of days for which the warning was issued was 54. However, these days include 22 instances when the temperature did not exceed 30 • C. Therefore, the number of days for which warnings were issued when the temperatures ranged between 30 • C and 33 • C was only 32. Approximately 84% of the days when the daily maximum temperature was between the morbidity threshold and the mortality threshold were missed. As a result, it is highly likely that warning caused by the threshold temperature of mortality is not effective in preventing heat-related morbidity. Table 1. Number of days on which warning was issued in Seoul, from 2011 to 2018 (based on the assumption that a warning was issued from two days before the daily maximum temperature reached the threshold temperature of mortality (33 • C)).

Type of Day
Number of Days 1 Number of days when the daily maximum temperature ≥30 • C and <33 • C 194 2 Number of days when the warning was issued 54 3 Number of days when the daily maximum temperature <30 • C in cases 2 22 4 Number of days missed from the warning in cases 1 ( 1 − ( 2 − 3 )) 162

Threshold Temperatures and Excess Mortality
Excess mortality tended to increase on average above the mortality threshold temperature. The average of excess mortality was 5.79 per day above 33 • C. On the other hand, the average of excess mortality was −0.02 between the daily maximum temperatures from 30 • C to 33 • C ( Table 2). The difference of the two average values was statistically significant (p-value = 0.0013). The 75th percentile of excess mortality above the mortality threshold temperature was about three times higher than the value above the morbidity threshold temperature. The maximum excess mortality (44.79) above the mortality threshold temperature was also approximately twice the value of the excess mortality (24.54) that occurred after the morbidity threshold temperature.
This result suggests that warnings based on the threshold temperature of heat-related morbidity overestimates the risk of mortality by heatwaves. Frequent issuance of warnings may reduce the effectiveness of the warning by increasing the fatigue of the consumer receiving the information. The graph comparing the daily excess mortality occurrence and the two threshold temperatures shows that the threshold temperature for heat-related morbidity has a limit to the warning of the risk of mortality. In Figure 5, the bar graph indicates the excess mortality. The red bars indicate the days when the number of deaths was greater than the expected deaths. Blue bars indicate fewer deaths than expected. The days when excess mortality increased dramatically appeared in 24 July, and between 1 and 4 August. During this period, the daily maximum temperature remained above 33 • C. It was difficult to find a definite increase in excess mortality within the temperatures between 30 • C and 33 • C. The threshold temperature of heat-related morbidity is difficult to use to warn of the risk of death from heatwaves. Int. J. Environ. Res. Public Health 2021, 18, x FOR PEER REVIEW 8 of 13

Performance Comparison of Heatwave Early Warnings
According to the comparison of actual and predicted HRI occurrences, the AUC was improved when the two threshold temperatures were combined compared to using the threshold temperature for either mortality or morbidity. Table 3 shows the evaluation results of warnings for Level-1 and Level-2 according to the type of threshold temperature. The actual events were HRI patients and deaths monitored in emergency departments. Level-1 was defined as the day on which the number of patients with HRI increased (50th to 90th percentile patients) to evaluate the performance of threshold temperature against morbidity. Level-2 was defined as the day on which deaths from HRI occurred or the number of patients with HRI increased to more than 90th percentile to evaluate the performance of the mortality threshold temperature.
The AUCs for the mortality threshold temperature were 0.60 and 0.86 at Level-1 and Level-2, respectively. When the morbidity threshold temperature was used, the AUC for Level-1 increased to 0.73, but the AUC decreased to 0.78 for Level-2. When the two threshold temperatures were combined, the AUCs were 0.74 and 0.86, respectively ( Table 3). The combination of the two threshold temperatures showed the best performance in the early warning of heatwaves.

Performance Comparison of Heatwave Early Warnings
According to the comparison of actual and predicted HRI occurrences, the AUC was improved when the two threshold temperatures were combined compared to using the threshold temperature for either mortality or morbidity. Table 3 shows the evaluation results of warnings for Level-1 and Level-2 according to the type of threshold temperature. The actual events were HRI patients and deaths monitored in emergency departments. Level-1 was defined as the day on which the number of patients with HRI increased (50th to 90th percentile patients) to evaluate the performance of threshold temperature against morbidity. Level-2 was defined as the day on which deaths from HRI occurred or the number of patients with HRI increased to more than 90th percentile to evaluate the performance of the mortality threshold temperature. The AUCs for the mortality threshold temperature were 0.60 and 0.86 at Level-1 and Level-2, respectively. When the morbidity threshold temperature was used, the AUC for Level-1 increased to 0.73, but the AUC decreased to 0.78 for Level-2. When the two threshold temperatures were combined, the AUCs were 0.74 and 0.86, respectively ( Table 3). The combination of the two threshold temperatures showed the best performance in the early warning of heatwaves.
The daily comparison of the actual and predicted events showed that the frequency of false negative (actual: yes, predicted: no) for Level-1 was high when the mortality threshold temperature was used. The prediction failure for Level-1 can be clearly seen in the 2016 and 2017 cases. In the case of using only the morbidity threshold temperature, there were many false positives (actual: no, predicted: yes) for Level-2. There was a pronounced tendency in 2016 and 2018 to overestimate Level-2. When both threshold temperatures were used, reasonable results were obtained in the prediction of Level-1 and Level-2 compared to when using a single threshold temperature ( Figure 6).
The daily comparison of the actual and predicted events showed that the frequency of false negative (actual: yes, predicted: no) for Level-1 was high when the mortality threshold temperature was used. The prediction failure for Level-1 can be clearly seen in the 2016 and 2017 cases. In the case of using only the morbidity threshold temperature, there were many false positives (actual: no, predicted: yes) for Level-2. There was a pronounced tendency in 2016 and 2018 to overestimate Level-2. When both threshold temperatures were used, reasonable results were obtained in the prediction of Level-1 and Level-2 compared to when using a single threshold temperature ( Figure 6).

Conclusions
This study showed the possibility that the warning system based on excess mortality may have limitations in reducing heat-related morbidity. The KMA is issuing warnings for heatwaves of Level-1 and Level-2, respectively, using threshold temperatures of 33 °C and 35 °C. According to our findings, these criteria are ineffective in the prevention of heat-related morbidity because the threshold temperatures are designed based on mortality. Most cases of patients with heat-related illness have already occurred before reaching Level-1. Furthermore, this study showed that the warning system based on heat-related morbidity would have limitations when warning about excessive mortality. The reason can be found in the difference in threshold temperature for morbidity and mortality.
This study suggests that, for the heatwave warning system to contribute to reducing both patient admissions and casualties, it is necessary to combine the threshold temperature of morbidity and mortality. The combination of morbidity and mortality thresholds showed improved performance compared to a single threshold temperature. Therefore, it is possible to improve the effectiveness of the warning system by identifying the lower (higher) threshold based epidemiological studies of morbidity (mortality).

Conclusions
This study showed the possibility that the warning system based on excess mortality may have limitations in reducing heat-related morbidity. The KMA is issuing warnings for heatwaves of Level-1 and Level-2, respectively, using threshold temperatures of 33 • C and 35 • C. According to our findings, these criteria are ineffective in the prevention of heat-related morbidity because the threshold temperatures are designed based on mortality. Most cases of patients with heat-related illness have already occurred before reaching Level-1. Furthermore, this study showed that the warning system based on heat-related morbidity would have limitations when warning about excessive mortality. The reason can be found in the difference in threshold temperature for morbidity and mortality.
This study suggests that, for the heatwave warning system to contribute to reducing both patient admissions and casualties, it is necessary to combine the threshold temperature of morbidity and mortality. The combination of morbidity and mortality thresholds showed improved performance compared to a single threshold temperature. Therefore, it is possible to improve the effectiveness of the warning system by identifying the lower (higher) threshold based epidemiological studies of morbidity (mortality). This study has apparent limitations in that we only dealt with the case of Seoul. Evidence may need to be supplemented with more case studies in the future. Nevertheless, this study is significant because it raised and demonstrated issues of the heatwave warning system developed using a single health impact proxy. The results of this study can contribute to improving the current Heat Watch Warning System of Korea by providing warnings for morbidity as well as mortality.

Conflicts of Interest:
The authors declare no conflict of interest. Level 1: When maximum temperature is more than 33 • C for at least two consecutive days.

Appendix A
Level 2: When maximum temperature is more than 35 • C for at least two consecutive days.
Excess mortality [37][38][39]  Excessive Heat Outlooks: When the potential for an excessive heat event in the next 3-7 days is identified. Excessive Heat Watches: When conditions are favorable for an excessive heat event in the next 24 to 72 h. Heat Advisory: Within 12 h of the onset of extremely dangerous heat conditions. The general rule of thumb for this Advisory is when the maximum heat index temperature is expected to be 100 • F or higher for at least 2 days, and night time air temperatures will not drop below 75 • F; however, these criteria vary across the country, especially for areas that are not considered to exhibit dangerous heat conditions. Excessive Heat Warning: Within 12 h of the onset of extremely dangerous heat conditions. The general rule of thumb for this Warning is when the maximum heat index temperature is expected to be 105 • F or higher for at least 2 days and night time air temperatures will not drop below 75 • F; however, these criteria vary across the country, especially for areas not used to extreme heat conditions. Human experimentation [15]