7.1. Performance Verification of Multivariate Time Series Segmentation Based on MRHLP
To verify the effectiveness and robustness of the MRHLP algorithm in multivariate time series segmentation point detection, this algorithm was compared with six kinds of algorithms in the literature.
- (1)
MHMMR [
18] (Multiple hidden Markov model): An unsupervised multivariate time series segmentation model based on hidden Markov model regression.
- (2)
MPWR [
19] (multiple polynomial piecewise regression model): an optimal segmentation method for multivariate time series based on polynomial piecewise regression model.
- (3)
ACA [
20] (Aligned cluster analysis): The problem of time series segmentation is transformed into the problem of time series clustering, and a method of alignment clustering analysis based on dynamic time alignment kernel and dynamic programming was proposed.
- (4)
HACA [
21] (Hierarchical aligned cluster analysis): An unsupervised bottom-up hierarchical aligned cluster analysis method was proposed by combining the standard vector space clustering method, dynamic time alignment kernel, and dynamic programming.
- (5)
A segmentation method based on data point clustering: GMM [
22], SC, FCM [
23], and k-means clustering algorithms were used to directly cluster multivariate time series data, and discrete classification labels were used to segment time series data.
- (6)
KPCA-GG [
24]: A multivariate time series segmentation algorithm based on kernel principal component analysis and Gath Geva fuzzy clustering.
In essence, the problem of multivariate time series segmentation is to divide a long time series into segments of multivariate time series with different lengths and patterns. It can be regarded as a clustering problem of multivariate time series or a classification problem. In this paper, we regarded it as a classification problem and used accuracy, precision, recall, F1 score, specificity, and Gmean to evaluate the performance of each segmentation algorithm.
- (1)
Accuracy
- (2)
Precision
- (3)
Recall
- (4)
F1-score
- (4)
Specificity
- (5)
Gmean
The meaning of the above formula is shown in
Table 2.
7.1.1. Segmentation Performance Verification of Algorithm Based on Simulation Data
The advantage of using synthetic datasets to verify the segmentation performance of the algorithm is that the correct segmentation is known, which is convenient for evaluating its performance. In this paper, the first-order vector autoregression model [
25] was used to generate two sets of 10-dimensional time series data sets. The number of data points is 1000, including eight and six modes, respectively. The specific model expressions are as follows:
where
is a multivariate Gaussian with mean 0 and variance
.
Two multivariate time series were generated based on Equation (88), and the generated multivariate time series is shown in
Figure 9. The details are as follows:
- (1)
Generate a 10-dimensional time series containing eight sub-segments, and the real segmentation positions are 100, 220, 420, 520, 670, 800, 920, and 1000.
- (2)
Generate a 10-dimensional time series containing six sub-segments, and the real segmentation positions are 150, 350, 500, 700, 850, and 1000.
Figure 10 presents the segmentation results for two sets of synthetic data under different algorithms. As can be seen from
Figure 10, on the two sets of synthetic multivariate time series, the proposed MRHLP algorithm can achieve more accurate segmentation results, which are relatively close to the real segmentation position, indicating the effectiveness of the MRHLP algorithm in handling multivariate time series segmentation problems.
To better verify the performance of the algorithm, the MRHLP multivariate time series segmentation algorithm proposed in this paper was compared with KPCA-GG, GMM, SC, k-means, FCM, MHMMR, MPWR, ACA, and HACA in terms of average Precision, Recall, Accuracy, Specificity, F1-score, and Gmean under the two synthetic datasets. The performance comparison results are shown in
Table 3 and
Table 4. It can be seen that, on the synthetic multivariate time series 1, the MRHLP algorithm can segment it very accurately, while the segmentation results obtained by other algorithms have certain errors. On the synthetic multivariate time series 2, the segmentation performance indicators of the MRHLP algorithm are close to 1, and the performances of the other algorithms are biased. In conclusion, the algorithm proposed in this paper has the best performance, can relatively accurately segment the multivariate time series, and achieve a segmentation result closer to the real segmentation situation.
7.1.2. Segmentation Performance Verification of Algorithm Based on Air Combat Situation Data
To verify the feasibility and effectiveness of the MRHLP multivariate time series segmentation algorithm in solving the problem of extracting medium-distance air combat situation primitives, two groups of air combat trajectory data as shown in
Figure 11 were selected from the air combat training system to test the performance of the algorithm, and the algorithm was compared with six algorithms in the literature.
Figure 12 shows the segmentation results of two groups of real air combat trajectory data under different algorithms. It can be seen that the MRHLP algorithm proposed in this paper can obtain more accurate segmentation results on two groups of real air combat trajectory data, which is closer to the real segmentation position, indicating that the MRHLP algorithm can effectively deal with the problem of time series segmentation of air combat situation characteristic parameters, and accurately divide the air combat confrontation process into different air combat situation primitives.
To better verify the performance of the algorithm, the MRHLP multivariate time series segmentation algorithm proposed in this paper was compared with KPCA-GG, GMM, SC, k-means, FCM, MHMMR, MPWR, ACA, and HACA in terms of average Precision, Recall, Accuracy, Specificity, F1 score and Gmean under two groups of real air combat trajectory data. The performance comparison results are shown in
Table 5 and
Table 6.
On the two groups of real air combat trajectory data, the MRHLP algorithm performs better than the other nine algorithms in multivariate time series segmentation. It can relatively accurately segment the time series of air combat situation characteristic parameters and can obtain trajectory fragments reflecting different air combat situations, providing a reliable data basis for the optimization of evaluation weights.
7.2. Verification of Air Combat Situation Prediction Model Based on DWOSELM
To verify the effectiveness and advancement of the algorithm proposed in this paper, OSELM [
26], OSKELM [
27], online regularized time series prediction algorithms (i.e., OSRELM [
28], OSRKELM [
29], RR-OSELM [
30]), and online regularized time series forecasting algorithms with forgetting factor (i.e., FP-OSELM [
31], DFF-OSELM [
32], FFOS-RKELM [
33], FGR-OSELM [
34]) were utilized for contrast. The parameter settings of all algorithms are shown in
Table 7 and the number of neurons in the hidden layer of all ELMs was set to 20.
To better compare performances of different algorithms, four measurements, i.e., root mean square error (RMSE), mean absolute deviation, (MAD), mean absolute percentage error (MAPE), and normalized mean square error (NMSE) were adopted in our work to evaluate the prediction accuracies. Mathematical expressions of RMSE, MAD, MAPE, and NMSE are presented as follows:
where
is the actual value,
is the predicted value, and
is the average of
.
According to the description of relative situation information of the enemy and us in
Section 2, based on the target position information obtained by the airborne radar sensor, the speed (value and angle), flight path angle, and heading angle of the target can be obtained by solving. On this basis, the target azimuth angle and entry angle can be further obtained. Therefore, the DWOSELM model can be used to accurately predict the maneuvering trajectory of the target and then the future air combat situation characteristic information of the target can be obtained through calculation. In this paper, a set of single machine confrontation trajectory data was extracted from the confrontation training system and used to verify the prediction performance of the DWOSELM model. The trajectory of the air combat confrontation is shown in
Figure 13.
To reduce the contingency error of the prediction algorithm, the prediction result of each algorithm is the average result obtained from conducting 20 repeated experiments. The results of the single-step prediction of the target maneuver trajectories by the different algorithms are shown in
Table 8. To more intuitively compare the prediction performance of different algorithms on the target maneuver trajectory, the absolute error values of single-step prediction of different algorithms are presented in the form of a figure, as shown in
Figure 14.
Table 8 shows the prediction results of each algorithm on the target maneuver trajectory and the single-step prediction error of each algorithm is shown in
Figure 14. It can be seen that, in terms of algorithm prediction accuracy, the DWOSELM prediction algorithm proposed in this paper has the smallest RMSE, MAD, MAPE, and NMSE. Under the same test data set, hardware platform, and initial parameter settings of the algorithm, its prediction accuracy is significantly better than that of other prediction methods in the literature. In terms of algorithm running time, the DWOSELM algorithm needs a little more running time due to the introduction of the sample dynamic weighting strategy, but it still meets the real-time requirements of the algorithm in the air combat decision-making process. To sum up, the air combat target maneuver trajectory prediction model based on the DWOSELM algorithm can accurately obtain the target’s future state information in real time, and then the future air combat situation information can be obtained.
7.3. Performance Verification of Air Combat Situation Assessment Model
To verify the performance of the DSASKEWO air combat situation assessment model proposed in this paper, this assessment model and the situation assessment based on missile attack area (SAMAA) model in [
35] were, respectively, applied to medium-range air combat maneuver decision-making to test their effectiveness. The specific implementation scheme of the medium-range air combat maneuver decision-making method was as follows:
Step1: Based on the situation information between the enemy and us at the current time t, execute the control commands of all actions in the action library respectively and conduct maneuver exploration.
Step2: Predict all possible positions of our aircraft in the next stage through step1, solve the air combat situation between us and the enemy at each position, and obtain the air combat situation set at the next moment. A represents the number of maneuver actions, and the membership value set of the situation parameter variables corresponding to all maneuver actions.
Step3: Calculate the mean value and standard deviation of the set corresponding to the maneuver, obtain the binary array composed of and , form the set , select the largest expected element in , and take its corresponding maneuver action as the tactical action to be executed.
Step4: Time update, return to step1.
To fully illustrate the effectiveness of the air combat situation assessment method proposed in this paper, the DSASKEWO assessment method and the SAMAA assessment method were used, respectively, for the simple maneuver and some complex maneuvers of the enemy aircraft, and the air combat maneuver decision simulation experiment was conducted by combining the above maneuver decision methods. The air combat simulation experiment scenario is shown in
Table 9.
- (1)
Scenario 1
Scenario setting: The enemy aircraft is in the rear position of our fighter. The enemy aircraft adopts a constant speed maneuver, which initially forms the dominant position of the enemy.
When we and the enemy formed the initial air combat situation of tail-chasing, different air combat situation assessment methods were used to conduct maneuver decision simulation experiments under the same initial conditions. The confrontation simulation results, the change curve of each situation advantage function value, the change curve of the comprehensive situation advantage function value, and the change curve of the situation advantage function index weight are shown in
Figure 15,
Figure 16,
Figure 17 and
Figure 18.
From the maneuver decision results shown in
Figure 15, it can be seen that, to allow our fighter to obtain a greater attack-occupying advantage, the maneuver decision methods based on the two situation assessment models adopted a series of maneuver strategies to guide the fighter to form a tail-chasing situation relative to the enemy. The maneuver decision-making model based on SAMAA adopts the maneuver sequence of “right turn-climb-right dive-right turn-right climb-left dive-right turn-keep flight” to change the air combat situation from a disadvantage to an advantage, to obtain the occupying advantage against the enemy. However, the maneuver decision model based on DSASKEWO adopts a relatively simple maneuver sequence, which is “left dive-direct flight-left turn-direct flight”, and finally realizes the transformation from disadvantage to advantage. It shows that maneuver decision-making based on the air combat situation assessment model proposed in this paper can guide our fighter to complete maneuver dominance more quickly and accurately, thus accelerating the operational cycle of “Observation, Orientation, Decision, Action”, namely the OODA loop.
Through the analysis of the comprehensive situation advantage function and the change curve of each situation advantage function value, it can be seen that the changing trend of each parameter based on the DSASKEWO maneuver decision model is unchanged. Therefore, the load borne by the pilot is also small, which is conducive to the pilot controlling the fighter for air combat confrontation while reducing energy loss, avoiding unnecessary maneuvers, and being more conducive to our air combat victory.
To sum up, it can be seen that the performance of the air combat situation assessment model proposed in this paper is better. When our aircraft is in an unfavorable situation, it can quickly and accurately guide the carrier to maneuver to occupy a favorable situation.
- (2)
Scenario 2
Scenario setting: The fighter meets the enemy in the front hemisphere and the enemy aircraft adopts snake maneuver strategy to form an initial situation of mutual disadvantage.
To fully verify the effectiveness of the air combat situation assessment method proposed in this paper, the maneuver decision verification experiment was conducted under the condition that the enemy conducts complex maneuvers. Assuming that the enemy aircraft performs a snake maneuver, under the initial situation of mutual disadvantage between us and enemy, different air combat situation assessment methods were used to conduct maneuver decision simulation experiments under the same initial conditions. The results of the confrontation simulation, the change curve of each situation advantage function value, the change curve of comprehensive situation advantage function value, and the change curve of situation advantage function index weight are shown in
Figure 19,
Figure 20,
Figure 21 and
Figure 22.
It can be seen from the two maneuver decision results shown in
Figure 19 that, for the snake maneuver adopted by the enemy, the maneuver decision model based on SAMAA adopted the maneuver sequence of “left climb-left dive-right turn-left turn”, while the maneuver decision model based on DSASKEWO adopted the maneuver sequence of “left climb-flat flight”. It is obvious that the maneuver decision-making method based on the situation assessment model proposed in this paper can better deal with the maneuver strategy adopted by the target. According to the maneuver decision sequence, in the initial stage of decision-making, both methods perform left climbing maneuvers but the execution time is different. The SAMAA-based maneuver decision-making method takes too long to execute the left climb, which causes our aircraft to miss the opportunity to form a tail-chasing dominant situation, reducing the decision-making efficiency.
According to the change curves of the comprehensive situation advantage function values and the change curves of each situation advantage function values shown in
Figure 20 and
Figure 21, the maneuver decision method based on the DSASKEWO evaluation model is highly volatile, while the maneuver decision method based on the DSASKEWO evaluation model is relatively stable; especially in the later stage of the confrontation, the comprehensive situation advantage is stable and has obvious advantages. This fully demonstrates the rationality and effectiveness of the maneuver decision-making method based on the DSASKEWO evaluation model.
To sum up, when the enemy aircraft adopts complex maneuvers, the maneuver decision-making method based on the DSASKEWO evaluation model is more reasonable and effective, and can quickly and accurately guide the fighter to complete maneuver occupation and form an occupation advantage against the enemy.