The Accuracy of Real-Time h_mF2 Estimation from Ionosondes

Abstract

A total of 4991 ionograms recorded from April 1997 to December 2017 by the Millstone Hill Digisonde (42.6°N, 288.5°E) were considered, with simultaneous N_e(h)_[ISR] profiles recorded by the co-located Incoherent Scatter Radar (ISR). The entire ionogram dataset was scaled with both the Autoscala and ARTIST programs. The reliability of the h_mF2 values obtained by ARTIST and Autoscala was assessed using the corresponding ISR values as a reference. Average errors Δ and the root mean square errors RMSE were computed for the whole dataset. Data analysis shows that both the Autoscala and ARTIST systems tend to underestimate h_mF2 values with |Δ| in all cases less than 10 km. For high magnetic activity ARTIST offers better accuracy than Autoscala, as evidenced by RMSE_[ARTIST] < RMSE_[Autoscala], under both daytime and nighttime conditions, and considering all hours of the day. Conversely, under low and medium magnetic activity Autoscala tends to estimate h_mF2 more accurately than the ARTIST system for both daytime and nighttime conditions, when RMSE_[Autoscala] < RMSE_[ARTIST]. However, RMSE_[Autoscala] slightly exceeds RMSE_[ARTIST] for the day as a whole. RMSE values are generally substantial (RMSE > 16 km in all cases), which places a limit on the results obtainable with real-time models that ingest ionosonde data.

1. Introduction

The ionosphere is a highly variable medium affecting HF radio propagation, which is used in long-distance communication and detection. Mean climatological conditions are useful to determine a “base level” for the design and operation of HF systems. h_mF2 is one of the important variables determining these conditions, and its importance also lies in its predicted decrease as one of the main ionospheric effects of the increasing concentrations of greenhouse gases. Its value is traditionally estimated by means of simple empirical formulations using the M(3000)F2 factor scaled from ionosondes [1] or through more sophisticated expressions involving additional characteristics (see e.g., [2,3], and references therein). The importance of h_mF2 has also stimulated studies of the effects of different modeling decisions in the International Reference Ionosphere model (IRI-2016) [4,5]. The various long-term models available today are based on ionosonde data with h_mF2 obtained from the vertical electronic density profile N_e(h), which in turn is derived from the ionogram. This was first achieved applying a polynomial inversion method that required the intervention of an operator [6], subsequently being automated [7,8,9].

However, long-term models are unable to effectively forecast ionospheric variability because h_mF2 varies not only due to thermospheric conditions but also in response to dynamic processes in the upper atmosphere. At mid-latitudes, field-aligned diffusion and recombination losses determine the h_mF2 in the absence of active vertical drift. Vertical drifts displace h_mF2 to a new equilibrium position in conjunction with the field-aligned redistribution of the plasma [10]. Therefore, real-time measurements are required as model inputs. The models are mostly empirical and among these the International Reference Ionosphere–Real Time Assimilative Mapping (IRI–RTAM) approach is particularly promising. It ingests data from the Global Ionospheric Radio Observatory (GIRO) [11] to adapt the IRI’s empirical background maps of ionospheric characteristics to match the observations [12]. A new Australian regional h_mF2 forecast model was also recently developed using ionosonde measurements and the bidirectional Long Short-Term Memory (bi-LSTM) method. This model predicts an h_mF2 value for the next hour based on data for the last five hours at the same location [13]. Even physical models, like SAMI2-CNU (the Chungnam National University (Daejeon, South Korea) in-house revised version of the open source 2-dimensional Another Model of the Ionosphere (SAMI2) developed at the Naval Research Laboratory (Washington, D.C.)), can be used as nowcast models for the regional mid-latitude ionosphere by assimilating ionosonde data in near-real time [14].

h_mF2 is also used as an input parameter in a new method to retrieve neutral temperature T_n and composition [O], [N₂], [O₂] from N_e(h) in the daytime mid-latitude F2 region under both quiet and disturbed conditions. Possible factors that can influence the achieved accuracy have been investigated. Several tests have demonstrated that discrepancies in the h_mF2 values provided by Digisondes [15,16] could have a significant impact on the performance of this method [17].

Serious concern about global warming of the troposphere has generated widespread interest in the study of long-term trends in the ionosphere since the early 1990s. Some research has linked ionospheric trends to anthropogenic sources, like the increase in greenhouse gas concentrations, while other studies identify natural causes, such as long-term changes in solar and geomagnetic activity, and secular variations in the Earth’s main magnetic field [18]. Long-term h_mF2 trends have been specifically studied in several works, making use of the data available in international databases (e.g., [19]).

In this work we consider the accuracy achieved in h_mF2 estimation by two automatic systems for ionogram interpretation: ARTIST [7,8,9], and Autoscala [20,21]. These data feed real-time ionospheric models and affect their performance. Furthermore, the same data populate international databases and constitute the measurements on which future retrospective studies will be based, at a historical moment when the huge resources necessary for manual data validation are often lacking.

2. Materials and Methods

The present assessment considered 4991 ionograms recorded in the period from April 1997 to December 2017 by the Digisonde [22] installed at Millstone Hill (42.6°N, 288.5°E), together with simultaneous N_e(h)_[ISR] profiles recorded by the co-located Incoherent Scatter Radar (ISR). The ionograms included were those for which the critical frequency f_oF2 provided by Autoscala and ARTIST matched the ISR observations within 0.1 MHz (in line with International Union of Radio Science (URSI) standard [23]), in order to avoid influencing the analysis with cases of incorrect f_oF2 autoscaling. The average errors:

$${\Delta }_{\left[\mathrm{ionosonde}\right]}={{\displaystyle \sum }}_{i=1}^N\frac{h_{\mathrm{m}}\mathrm{F}{2}_{\left[\mathrm{ionosonde}\right]i}-h_{\mathrm{m}}\mathrm{F}{2}_{\left[\mathrm{ISR}\right]i}}{N},$$

(1)

and the root mean square errors:

$${\mathrm{RMSE}}_{\left[\mathrm{ionosonde}\right]}=\sqrt{{{\displaystyle \sum }}_{i=1}^N\frac{{\left(h_{\mathrm{m}}\mathrm{F}{2}_{\left[\mathrm{ionosonde}\right]i}-h_{\mathrm{m}}\mathrm{F}{2}_{\left[\mathrm{ISR}\right]i}\right)}^2}{N}},$$

(2)

with the symbols being self-evident in meaning, were computed for the h_mF2 data across the whole dataset, where both the Autoscala and ARTIST systems were used to scale h_mF2_[ionosonde]. An error in h_mF2_[ISR] determination can be assumed around ±10 km at Millstone Hill ISR [24].

The study considered nocturnal (between 22:00 and 02:00 local time (LT)) and diurnal (between 10:00 and 14:00 LT) conditions separately, and under high, medium, and quiet geomagnetic conditions. Magnetic activity is classed as disturbed if a magnetic index value a_p > 40 was observed over the previous 24 h, moderately disturbed if 7 < a_p ≤ 40 was observed over the previous 24 h, and quiet if a_p ≤ 7 was constant over the previous 24 h.

The Student’s t-test for the paired (h_mF2_[ionosonde]; h_mF2_[ISR]) data sets was also performed, for both the Autoscala and ARTIST systems. The aim of the test is to verify whether the mean difference between two data sets is statistically significant [25]. This information is given by the significance level, i.e., maximum probability p_[ionosonde] that Δ_[ionosonde] is not significant. In other words, a significance level of p_[ionosonde] means that there is a lower probability than p_[ionosonde] that the null hypothesis E[h_mF2_[ionosonde] − h_mF2_[ISR]] = 0 is true. The null hypothesis is rejected when p ≤ 0.05. In this case it is assumed that E[h_mF2_[ionosonde] − h_mF2_[ISR]] ≠ 0 and Δ_[ionosonde] is statistically significant.

3. Results

The results obtained are shown in

Table 1. The average errors (Δ_[Autoscala], Δ_[ARTIST]), the number of measurements considered, the Student t-test results, which provides an indication of the statistical significance p of the observed differences, and the root mean square errors (RMSE_[Autoscala], RMSE_[ARTIST]), at nighttime (22:00–02:00 LT), daytime (10:00–14:00 LT), and for the day as a whole. Significant cases are highlighted with bold. The analysis was repeated for magnetically strongly disturbed (upper table), moderately disturbed (middle table), and quiet (bottom table) conditions. Conditions were classed as strongly disturbed if a magnetic index value a_p > 40 was observed over the previous 24 h, moderately disturbed if 7 < a_p ≤ 40 was observed over the previous 24 h, and quiet if a_p ≤ 7 was observed over the previous 24 h.

	High Magnetic Activity
	Δ[Autoscala] (km)	Δ[ARTIST] (km)	RMSE[Autoscala] (km)	RMSE[ARTIST] (km)	N° of cases	p[Autoscala]	p[ARTIST]
Night	−1.57	5.27	34.09	19.53	95	0.65622	0.00782
Day	−1.32	−3.02	21.65	16.41	100	0.54467	0.06524
Night and day	−3.26	−2.58	24.79	20.91	472	0.00418	0.00730
	Medium Magnetic Activity
	Δ[Autoscala] (km)	Δ[ARTIST] (km)	RMSE[Autoscala] (km)	RMSE[ARTIST] (km)	N° of cases	p[Autoscala]	p[ARTIST]
Night	−6.45	−7.28	25.90	26.37	335	3.9·10−6	2.8·10−7
Day	−3.00	−8.7	19.71	20.79	768	2.3·10−5	6.0·10−34
Night and day	−4.95	−8.81	24.93	21.99	2611	1.3·10−24	3.1·10−101
	Low Magnetic Activity
	Δ[Autoscala] (km)	Δ[ARTIST] (km)	RMSE[Autoscala] (km)	RMSE[ARTIST] (km)	N° of cases	p[Autoscala]	p[ARTIST]
Night	−5.36	−2.02	17.77	24.84	171	5.8·10−5	0.28876
Day	3.09	−4.16	16.63	17.05	583	6.2·10−6	2.2·10−9
Night and day	−1.85	−5.81	20.07	18.32	1908	5.5·10−5	8.0·10−46

, along with the Student’s t-test results. Column 2 (3) reports the average error Δ_[Autoscala] (Δ_[ARTIST]), while column 4 (5) reports the root mean square error RMSE_[Autoscala] (RMSE_[ARTIST]). In column 6 the number of measurements considered is shown, while column 7 (8) reports the significance level given by the Student’s t-test. Cases of statistical significance of Δ_[Autoscala] (Δ_[ARTIST]) are highlighted with bold in column 2 (3).

The results are also presented in the form of histograms in Figure 1, Figure 2 and Figure 3, where the occurrence of different values of the differences h_mF2_[ionosonde] − h_mF2_[ISR] are shown for different hours of the day and different magnetic activity levels.

4. Discussion and Conclusions

The data reported in Table 1 show that both the Autoscala and ARTIST systems tend to underestimate h_mF2 values compared to ISR measurements, with a mean deviation in all cases of less than 10 km. This is a little better than the result obtained for the ARTIST system by Chen et al. [26], who estimated average peak height differences between −4 km (in winter) and −17 km (in summer), in a comparison of some 2000 profiles recorded at Millstone Hill in 1990. In the present study, mean overestimation was instead observed for ARTIST (Δ_[ARTIST] = +5.27 km) during nighttime under high magnetic activity, and for Autoscala (Δ_[Autoscala] = +3.09 km) during daytime under low magnetic activity.

Under high magnetic activity ARTIST shows better accuracy than Autoscala, as demonstrated by RMSE_[ARTIST] < RMSE_[Autoscala], in both nighttime and daytime conditions, and considering all hours of the day. Besides, the behavior of Autoscala during nighttime is particularly critical, due to high magnetic activity. Despite the very low mean underestimations of h_mF2_[Autoscala] compared to h_mF2_[ISR] (Δ_[Autoscala] = −1.57 km), clearly in these circumstances RMSE_[Autoscala] (equal to 34.06 km) is much higher than at other times. Under these conditions, there are also 7.37% of cases in which h_mF2_[Autoscala] − h_mF2_[ISR] > 40 km, while for ARTIST (RMSE_[ARTIST] = 19.52 km) the percentage is only 2.11%. Examples of such cases are shown in Figure 4a,b and Figure 5a,b, while Figure 4c and Figure 5c report the corresponding comparisons between the N_e(h) values provided by Autoscala, ARTIST, and from ISR data. These critical cases included some in which the trace was difficult to locate because of Spread-F conditions (see e.g., [27,28]) and Autoscala fails to correctly detect the trace (see Figure 4a,b). Conversely, in other critical cases the trace appears to have been correctly detected by both programs (see Figure 5a,b), and the greater accuracy achieved by ARTIST is probably linked to its more efficient estimation of the F-region semi-thickness parameter B₀ (see e.g., [29]), which describes the profile in the F2 region. However, among the 7 cases in which h_mF2_[Autoscala] − h_mF2_[ISR] > 40 km, there are 2 in which h_mF2_[ARTIST] also overestimates h_mF2_[ISR] by over 40 km.

Under low and medium magnetic activity Autoscala tends to estimate h_mF2 more accurately than the ARTIST system for individual hours during both daytime and nighttime conditions, when RMSE_[Autoscala] < RMSE_[ARTIST]. However, RMSE_[Autoscala] slightly exceeds RMSE_[ARTIST] for the day as a whole when all cases are considered. This means that close to the solar terminators, the accuracy of Autoscala’s h_mF2 tends to decline more than ARTIST’s. In spite of this, |Δ_[Autoscala]| < |Δ_[ARTIST]| in almost all cases, suggesting that Autoscala h_mF2 values tend to be closer to the real ones, even under high magnetic activity conditions.

In conclusion, the present work demonstrates a low systematic error in the determination of h_mF2 by ionosondes, with Δ < 10 km in all cases. The RMSE values, however, differ according to the various situations considered, but are generally higher, with RMSE > 16 km in all cases. This represents a limitation to the results obtainable from real-time models that ingest ionosonde data.