The Accuracy of Real-Time hmF2 Estimation from Ionosondes

A total of 4991 ionograms recorded from April 1997 to December 2017 by the Millstone Hill Digisonde (42.6◦N, 288.5◦E) were considered, with simultaneous Ne(h)[ISR] profiles recorded by the co-located Incoherent Scatter Radar (ISR). The entire ionogram dataset was scaled with both the Autoscala and ARTIST programs. The reliability of the hmF2 values obtained by ARTIST and Autoscala was assessed using the corresponding ISR values as a reference. Average errors ∆ and the root mean square errors RMSE were computed for the whole dataset. Data analysis shows that both the Autoscala and ARTIST systems tend to underestimate hmF2 values with |∆| in all cases less than 10 km. For high magnetic activity ARTIST offers better accuracy than Autoscala, as evidenced by RMSE[ARTIST] < RMSE[Autoscala], under both daytime and nighttime conditions, and considering all hours of the day. Conversely, under low and medium magnetic activity Autoscala tends to estimate hmF2 more accurately than the ARTIST system for both daytime and nighttime conditions, when RMSE[Autoscala] < RMSE[ARTIST]. However, RMSE[Autoscala] slightly exceeds RMSE[ARTIST] for the day as a whole. RMSE values are generally substantial (RMSE > 16 km in all cases), which places a limit on the results obtainable with real-time models that ingest ionosonde data.


Introduction
The ionosphere is a highly variable medium affecting HF radio propagation, which is used in long-distance communication and detection. Mean climatological conditions are useful to determine a "base level" for the design and operation of HF systems. h m F2 is one of the important variables determining these conditions, and its importance also lies in its predicted decrease as one of the main ionospheric effects of the increasing concentrations of greenhouse gases. Its value is traditionally estimated by means of simple empirical formulations using the M(3000)F2 factor scaled from ionosondes [1] or through more sophisticated expressions involving additional characteristics (see e.g., [2,3], and references therein). The importance of h m F2 has also stimulated studies of the effects of different modeling decisions in the International Reference Ionosphere model (IRI-2016) [4,5]. The various long-term models available today are based on ionosonde data with h m F2 obtained from the vertical electronic density profile N e (h), which in turn is derived from the ionogram. This was first achieved applying a polynomial inversion method that required the intervention of an operator [6], subsequently being automated [7][8][9].
However, long-term models are unable to effectively forecast ionospheric variability because h m F2 varies not only due to thermospheric conditions but also in response to dynamic processes in the upper atmosphere. At mid-latitudes, field-aligned diffusion and recombination losses determine the h m F2 in the absence of active vertical drift. Vertical drifts displace h m F2 to a new equilibrium position in conjunction with the field-aligned redistribution of the plasma [10]. Therefore, real-time measurements are required as model inputs. The models are mostly empirical and among these the International Reference Ionosphere-Real Time Assimilative Mapping (IRI-RTAM) approach is particularly promising. It ingests data from the Global Ionospheric Radio Observatory (GIRO) [11] to adapt the IRI's empirical background maps of ionospheric characteristics to match the observations [12]. A new Australian regional h m F2 forecast model was also recently developed using ionosonde measurements and the bidirectional Long Short-Term Memory (bi-LSTM) method. This model predicts an h m F2 value for the next hour based on data for the last five hours at the same location [13]. Even physical models, like SAMI2-CNU (the Chungnam National University (Daejeon, South Korea) in-house revised version of the open source 2-dimensional Another Model of the Ionosphere (SAMI2) developed at the Naval Research Laboratory (Washington, D.C.)), can be used as nowcast models for the regional mid-latitude ionosphere by assimilating ionosonde data in near-real time [14].
h m F2 is also used as an input parameter in a new method to retrieve neutral temperature T n and composition [O] [15,16] could have a significant impact on the performance of this method [17].
Serious concern about global warming of the troposphere has generated widespread interest in the study of long-term trends in the ionosphere since the early 1990s. Some research has linked ionospheric trends to anthropogenic sources, like the increase in greenhouse gas concentrations, while other studies identify natural causes, such as long-term changes in solar and geomagnetic activity, and secular variations in the Earth's main magnetic field [18]. Long-term h m F2 trends have been specifically studied in several works, making use of the data available in international databases (e.g., [19]).
In this work we consider the accuracy achieved in h m F2 estimation by two automatic systems for ionogram interpretation: ARTIST [7][8][9], and Autoscala [20,21]. These data feed real-time ionospheric models and affect their performance. Furthermore, the same data populate international databases and constitute the measurements on which future retrospective studies will be based, at a historical moment when the huge resources necessary for manual data validation are often lacking.

Materials and Methods
The present assessment considered 4991 ionograms recorded in the period from April 1997 to December 2017 by the Digisonde [22] installed at Millstone Hill (42.6 • N, 288.5 • E), together with simultaneous N e (h) [ISR] profiles recorded by the co-located Incoherent Scatter Radar (ISR). The ionograms included were those for which the critical frequency f o F2 provided by Autoscala and ARTIST matched the ISR observations within 0.1 MHz (in line with International Union of Radio Science (URSI) standard [23]), in order to avoid influencing the analysis with cases of incorrect f o F2 autoscaling. The average errors: and the root mean square errors: with the symbols being self-evident in meaning, were computed for the h m F2 data across the whole dataset, where both the Autoscala and ARTIST systems were used to scale h m F2 [ionosonde] . An error in h m F2 [ISR] determination can be assumed around ±10 km at Millstone Hill ISR [24]. The study considered nocturnal (between 22:00 and 02:00 local time (LT)) and diurnal (between 10:00 and 14:00 LT) conditions separately, and under high, medium, and quiet geomagnetic conditions. Magnetic activity is classed as disturbed if a magnetic index value a p > 40 was observed over the previous 24 h, moderately disturbed if 7 < a p ≤ 40 was observed over the previous 24 h, and quiet if a p ≤ 7 was constant over the previous 24 h.
The Student's t-test for the paired (h m F2 [ionosonde] ; h m F2 [ISR] ) data sets was also performed, for both the Autoscala and ARTIST systems. The aim of the test is to verify whether the mean difference between two data sets is statistically significant [25]. This information is given by the significance level, i.e., maximum probability p [ionosonde] that ∆ [ionosonde] is not significant. In other words, a significance level of p [ionosonde] means that there is a lower probability than p [ionosonde] that the null hypothesis [ISR] ] 0 and ∆ [ionosonde] is statistically significant.

Results
The results obtained are shown in Table 1  The results are also presented in the form of histograms in Figures 1-3, where the occurrence of different values of the differences h m F2 [ionosonde] − h m F2 [ISR] are shown for different hours of the day and different magnetic activity levels.

Discussion and Conclusions
The data reported in Table 1 show that both the Autoscala and ARTIST systems tend to underestimate hmF2 values compared to ISR measurements, with a mean deviation in all cases of less than 10 km. This is a little better than the result obtained for the ARTIST system by [26], who estimated average peak height differences between −4 km (in winter) and −17 km (in summer), in a comparison of some 2000 profiles recorded at Millstone Hill in 1990. In the present study, mean

Discussion and Conclusions
The data reported in Table 1 show that both the Autoscala and ARTIST systems tend to underestimate h m F2 values compared to ISR measurements, with a mean deviation in all cases of less than 10 km. This is a little better than the result obtained for the ARTIST system by Chen et al. [26], who estimated average peak height differences between −4 km (in winter) and −17 km (in summer), in a comparison of some 2000 profiles recorded at Millstone Hill in 1990. In the present study, mean overestimation was instead observed for ARTIST (∆ [ Figure 4a,b and Figure 5a,b, while Figures 4c and 5c report the corresponding comparisons between the N e (h) values provided by Autoscala, ARTIST, and from ISR data. These critical cases included some in which the trace was difficult to locate because of Spread-F conditions (see e.g., [27,28]) and Autoscala fails to correctly detect the trace (see Figure 4a,b). Conversely, in other critical cases the trace appears to have been correctly detected by both programs (see Figure 5a,b), and the greater accuracy achieved by ARTIST is probably linked to its more efficient estimation of the F-region semi-thickness parameter B 0 (see e.g., [29]), which describes the profile in the F2 region. However, among the 7 cases in which h m F2 [ Figure 4a,b and Figure 5a,b, while Figures 4c and 5c report the corresponding comparisons between the Ne(h) values provided by Autoscala, ARTIST, and from ISR data. These critical cases included some in which the trace was difficult to locate because of Spread-F conditions (see e.g., [27,28]) and Autoscala fails to correctly detect the trace (see Figure 4a,b). Conversely, in other critical cases the trace appears to have been correctly detected by both programs (see Figure 4a,b), and the greater accuracy achieved by ARTIST is probably linked to its more efficient estimation of the F-region semi-thickness parameter B0 (see e.g., [29]), which describes the profile in the F2 region.    Under low and medium magnetic activity Autoscala tends to estimate hmF2 more accurately than the ARTIST system for individual hours during both daytime and nighttime conditions, when RMSE[Autoscala] < RMSE[ARTIST]. However, RMSE[Autoscala] slightly exceeds RMSE[ARTIST] for the day as a whole when all cases are considered. This means that close to the solar terminators, the accuracy of Autoscala's hmF2 tends to decline more than ARTIST's. In spite of this, |Δ[Autoscala]| < |Δ[ARTIST]| in almost all cases, suggesting that Autoscala hmF2 values tend to be closer to the real ones, even under high magnetic activity conditions.
In conclusion, the present work demonstrates a low systematic error in the determination of hmF2 by ionosondes, with Δ < 10 km in all cases. The RMSE values, however, differ according to the various situations considered, but are generally higher, with RMSE > 16 km in all cases. This represents a limitation to the results obtainable from real-time models that ingest ionosonde data.  Under low and medium magnetic activity Autoscala tends to estimate h m F2 more accurately than the ARTIST system for individual hours during both daytime and nighttime conditions, when RMSE [Autoscala] < RMSE [ARTIST] . However, RMSE [Autoscala] slightly exceeds RMSE [ARTIST] for the day as a whole when all cases are considered. This means that close to the solar terminators, the accuracy of Autoscala's h m F2 tends to decline more than ARTIST's. In spite of this, |∆ [Autoscala] | < |∆ [ARTIST] | in almost all cases, suggesting that Autoscala h m F2 values tend to be closer to the real ones, even under high magnetic activity conditions.
In conclusion, the present work demonstrates a low systematic error in the determination of h m F2 by ionosondes, with ∆ < 10 km in all cases. The RMSE values, however, differ according to the various situations considered, but are generally higher, with RMSE > 16 km in all cases. This represents a limitation to the results obtainable from real-time models that ingest ionosonde data.