Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions

Abstract

This paper proposes a multimodal Transformer model that uses time-series data to detect and predict winter road surface conditions. For detecting or predicting road surface conditions, the previous approach focuses on the cooperative use of multiple modalities as inputs, e.g., images captured by fixed-point cameras (road surface images) and auxiliary data related to road surface conditions under simple modality integration. Although such an approach achieves performance improvement compared to the method using only images or auxiliary data, there is a demand for further consideration of the way to integrate heterogeneous modalities. The proposed method realizes a more effective modality integration using a cross-attention mechanism and time-series processing. Concretely, when integrating multiple modalities, feature compensation through mutual complementation between modalities is realized through a feature integration technique based on a cross-attention mechanism, and the representational ability of the integrated features is enhanced. In addition, by introducing time-series processing for the input data across several timesteps, it is possible to consider the temporal changes in the road surface conditions. Experiments are conducted for both detection and prediction tasks using data corresponding to the current winter condition and data corresponding to a few hours after the current winter condition, respectively. The experimental results verify the effectiveness of the proposed method for both tasks. In addition to the construction of the classification model for winter road surface conditions, we first attempt to visualize the classification results, especially the prediction results, through the image style transfer model as supplemental extended experiments on image generation at the end of the paper.

1. Introduction

In snow-covered and cold regions, which account for approximately 60% of the land area in Japan, numerous winter-related traffic accidents occur due to weather conditions, e.g., snowfall. Approximately 90% of these accidents are slip-related incidents associated with winter road surface conditions due to snow accumulation and ice formation [1]. In this context, road managers need to undertake snow and ice control operations, e.g., snow removal and the spreading of anti-freezing agents by detecting or predicting road surface conditions to prevent slip accidents [1,2].

Previous studies have investigated the detection or prediction of winter road surface conditions [3,4,5,6,7,8]. In the literature [3], the road surface condition was predicted based on the heat balance theory using digital geographical data, which represent the shape of the land, including roads on computers; however, this method requires the analysis of digital geographical data related to the road, and it is difficult to collect and accumulate such data for all roads. In another study [7], the automatic detection of winter road surface conditions was realized using deep learning models trained on images captured by vehicle-mounted cameras. Similarly, winter road surface conditions were classified using hierarchical deep learning models applied to images also captured by vehicle-mounted cameras [8]. Here, to use images captured by vehicle-mounted cameras, it is necessary to drive on the road to be analyzed with vehicles equipped with cameras. To reduce such efforts, in the literature [4], data obtained from sensors and fixed-point cameras installed along roads were adopted to detect or predict the winter road surface conditions using rule-based methods. In addition, a previous study [5] achieved detection by classifying road surface conditions using differential methods based on images captured by fixed-point cameras installed along the road (hereafter referred to as road surface images). However, due to the temporal variability of road surfaces and roadside features, methods based on differential approaches require manual updating of the reference images. Thus, there is a demand for models that can classify road surface conditions automatically and accurately to facilitate precise detection and prediction. Several studies have focused on the winter road surface condition classification using the images captured by vehicle-mounted cameras [9,10,11]. The purpose of these studies is to help with the construction of autonomous vehicles; however, our purpose is to assist road managers in reducing winter-related traffic accidents using fixed-point cameras.

The multimodal analysis, which uses several information sources, e.g., images and natural languages, has attracted significant attention for improving the representational ability of models [12,13,14,15]. For example, contrastive language image pre-training has been proposed as the pre-training framework for the multimodal analysis of vision and language [16]. Another example is to use the texts obtained from Twitter in addition to images for image sentiment analysis [17]. In this way, most works on multimodal analysis have used vision and language modalities; however, in the classification task of winter road surface conditions, the text information does not exist, and the other information is needed for multimodal analysis. Then, we previously proposed an automated classification method for road surface conditions using a multimodal multilayer perceptron (MLP) using images and auxiliary data [18]. Concretely, in that study, the features calculated from multiple modalities, including road surface images and auxiliary data related to the road surface conditions such as temperatures and traffic volume, were concatenated and input to the MLP to classify the road surface conditions. The cooperative use of multiple modalities allows for mutual complementation between modalities, and we improved classification accuracy compared to using a single modality. However, in the previous study, we focused on the construction of machine learning models using multiple modalities and performed multimodal analysis through a simple feature concatenation process. As a result, this approach may have inherent limitations in terms of classification accuracy. Thus, further improvements in classification accuracy can be expected by introducing the following processes.

• Time-series Analysis
  In the field of glaciology, a previous study [19] reported that snow accumulation extremes exhibit time-series variability. In addition, Hirai et al. [20] suggested that changes in road surface conditions are related to the transitions of these conditions over the past several timesteps. Thus, rather than relying on data from a single timestep (as in our previous study), using time-series data to classify road surface conditions is expected to improve the detection and prediction accuracy.
• Feature Integration using Attention Mechanisms
  In our previous study, feature integration was performed by concatenating the features derived separately from image and auxiliary data and then inputting them into an MLP. On the other hand, in the machine learning field, Transformers [21,22,23,24], which are the novel machine learning architecture focusing on the relationship of input data, have attracted significant attention for the remarkable performance based on the strong representational ability. With the advancement of such Transformers, recent research on feature integration has demonstrated that intermediate fusion, which combines features in the intermediate layers of neural networks using cross-attention, achieves higher accuracy than traditional feature integration methods [25,26,27,28,29]. Cross-attention is an attention mechanism [21] with several inputs, which facilitates the compensation of heterogeneous features calculated from multiple modalities. As a result, the cross-attention module enhances the representational ability after integration, and the use of feature integration based on cross-attention is expected to further improve classification accuracy.

In this paper, we propose a new method for classifying winter road surface conditions using a multimodal transformer (MMTransformer) capable of processing time-series data. In the proposed method, image and auxiliary features are extracted from data spanning multiple timesteps, and feature integration considering temporal changes is performed by applying cross-attention. With cross-attention, correlations are calculated feature-wise for input data across multiple timesteps, and attention is computed for each timestep. This procedure enables feature integration that accounts for temporal changes in road surface conditions. Finally, the classification of winter road surface conditions is realized using an MLP. By exploring methods for integrating multiple modalities and introducing time-series processing, we aim to achieve improvements in accuracy in the detection and prediction of road surface conditions.

In addition, the proposed method can learn the relationship between the input data and the corresponding teacher labels, which are the labels related to winter road surface conditions for training the model. By altering the teacher labels assigned to the input data during training, the proposed method can be adapted to both detection and prediction tasks. In experiments conducted on real-world data, we evaluated the effectiveness of the proposed method for both detection and prediction tasks with two sets of teacher labels. One experiment was conducted with the teacher labels being the road surface condition corresponding to the input data, and the subsequent experiment was conducted with the teacher labels being the road surface condition a few hours after the input data. This dual approach allows for a comprehensive assessment of the capabilities of the proposed method in detecting the current road surface conditions and predicting future road surface conditions.

In addition to the experiments on the classification of winter road surface conditions, we conducted supplemental extended experiments on image generation to visualize the classification results, particularly the prediction results in the Appendix A. To help road managers make decisions, it can be effective to incorporate classification results and road surface images that visualize the results. In this study, we generated such images using an image style transfer model conditioned by road surface conditions. Through these supplemental experiments and visualizing the transferred images, we confirmed the potential of the image transfer model for road surface images.

The primary contributions of this study are summarized as follows.

• A multimodal transformer model based on time-series processing and attention mechanisms is constructed to classify road surface conditions.
• Experiments conducted to evaluate the road surface condition detection and prediction tasks verify the effectiveness of the proposed classification model.
• The results of the supplemental extended experiments in the Appendix A demonstrate the potential of the image transfer model for road surface images.

The remainder of this paper is organized as follows. Section 2 introduces the data used in this study. The proposed method for the classification of winter road surface conditions is explained in Section 3. Then, the experimental results are reported in Section 4, and the supplemental extended experiments are discussed in Appendix A. Finally, Section 5 concludes the paper.

2. Data

In the following, we describe the data used in this study. We utilized road surface images acquired using fixed-point cameras and auxiliary data related to the road surface conditions. Specifically, these data were provided by the East Nippon Expressway Company Limited and were acquired from 2017 to 2019. The road surface images were captured at 20-min intervals from 1 December at 00:00 to 31 March at 23:40 each year. In addition, each road surface image was labeled with one of the following seven categories related to road surface conditions.

• Dry
  The road surface is free of snow, ice, and wetness.
• Wet
  The road surface is wet due to moisture.
• Black sherbet
  Tire tread marks are present, the snow contains a high amount of moisture, and the color of the road surface is black.
• White sherbet
  Tire tread marks are present, the snow contains a high amount of moisture, and the color of the road surface is white.
• Snow
  Snow has accumulated on the road surface, and the snow does not contain a high amount of moisture.
• Compacted snow
  There is no black shine and no tire tread marks.
• Ice
  Snow and ice are present on the road surface, and it appears black and shiny.

These labels were assigned by three experienced road managers, and they divided the annotation task and assigned the labels through visual inspections. Example road surface images for each category are shown in Figure 1, and the locations where the road surface images were captured are shown in Figure 2. Here, the image size is $640\times 480$ pixels. Please note that road surface images, including vehicles, were considered for analysis because the vehicles did not cover the entire road surface in the images.

Table 1. Auxiliary data and corresponding data types.

Data Content	Data Type
Location of road surface images	Discrete
Temperature	Continuous
Road temperature	Continuous
Amount of snowfall	Continuous
Traffic volume	Continuous
Average of vehicle speed	Continuous
Weather forecast six hours ago	Discrete
Temperature forecast six hours ago	Continuous
Snowfall forecast six hours ago	Continuous
Weather forecast 12 h ago	Discrete
Temperature forecast 12 h ago	Continuous
Snowfall forecast 12 h ago	Continuous
Weather forecast 18 h ago	Discrete
Temperature forecast 18 h ago	Continuous
Snowfall forecast 18 h ago	Continuous
Weather forecast 24 h ago	Discrete
Temperature forecast 24 h ago	Continuous
Snowfall forecast 24 h ago	Continuous

shows the contents of the auxiliary data and the corresponding data types. As shown in Table 1, the “location of road surface images” and “weather forecast” are discrete information, while other data contents are represented as continuous values. As shown in Figure 1 and Table 1, the images and auxiliary data differ significantly; thus, a feature integration mechanism is required to complement the deficiencies in each modality. Thus, we attempt to improve the classification accuracy of road surface conditions by integrating multiple modalities at several timesteps.

3. Classification of Winter Road Surface Conditions Using MMTransformer

In this section, we describe the proposed method to classify winter road surface conditions based on the MMTransformer, which can process time-series data using images and auxiliary data at multiple timesteps as inputs. First, we construct encoders for both the image and auxiliary data at each timestep to extract relevant features. We then calculate the integrated features with the characteristics of both the image and auxiliary data by performing feature integration based on cross-attention. Finally, by inputting the integrated features into an MLP, we can classify the winter road surface conditions. An overview and flowchart of the proposed method are shown in Figure 3 and Figure 4, respectively. Please note that the proposed model is trained in an end-to-end manner, which allows the image encoder to be fine-tuned and the parameters in the MLP to be optimized simultaneously. In the following, we explain the methods for feature extraction and feature integration based on cross-attention in Section 3.1 and Section 3.2, respectively.

3.1. Feature Extraction

Here, we describe the method employed to construct the encoders used to extract the features from the image and auxiliary data.

3.1.1. Visual Features

The proposed method utilizes output values from the intermediate layers of a pretrained deep learning model as visual features. For the deep learning model, we employ the Vision Transformer (ViT) [24] or its derivative methods [22,23], which have achieved high classification accuracy in image classification tasks. Training a model based on the ViT requires a large amount of training data; thus, we fine-tune a model pretrained on ImageNet [30] to extract the visual features with high representational ability from the road surface images.

In the ViT, as shown in Figure 5, patches obtained by dividing the images and position embeddings are input sequentially to linear layers and the Transformer encoder. The output values are calculated by the MLP head after the Transformer encoder. During fine-tuning of the ViT, transfer learning is performed on the Transformer encoder by replacing the MLP head. Specifically, in the proposed method, the visual feature ${\mathit{x}}_t^{\left(\mathrm{vis}\right)}\in {\mathbb{R}}^{d_{\mathrm{vis}}}$ for image ${\mathit{V}}_t$ at timestep t ($t=1,2,\dots ,T$, where T is the number of timesteps) is calculated as follows:

$$\begin{array}{c}\hfill {\mathit{x}}_t^{\left(\mathrm{vis}\right)}=f\left(E_{\mathrm{vis}}\left({\mathit{V}}_t\right)\right),\end{array}$$

(1)

where $E_{\mathrm{vis}}(·)$ is the pretrained Transformer encoder in the ViT-based model, and $f(·)$ is the MLP that calculates the visual features for input into the cross-attention mechanism. Thus, by employing an MLP head suitable for feature integration, it is possible to fine-tune the ViT-based model and train the cross-attention mechanism simultaneously.

3.1.2. Auxiliary Features

In the proposed method, the auxiliary data include both continuous quantitative variables, e.g., temperature and road temperature, and discrete qualitative variables, using nominal scales, e.g., location and weather conditions. Generally, in machine learning involving qualitative variables as inputs, one-hot encoding is used as a preprocessing method [31,32,33]. In one-hot encoding, elements equal to the number of items in the nominal scale are prepared, and the corresponding element is set to 1 (while others are set to 0). This procedure enables machine learning models to process qualitative variables. However, when one-hot encoded features ${\left\{x_i\right\}}_{i=0}^n$ are input to a neural network–based model, in the first layer of the forward propagation process, only the weights corresponding to the input elements with 1 are updated as follows:

$$\begin{array}{c}\hfill a_{01}=\sum _{i=0}^n{x}_i{W}_{i0}+b_0,\end{array}$$

(2)

where ${\left\{W_{i0}\right\}}_{i=0}^n$ represents the weights corresponding to $x_i$, and $a_{01}$ is the output value at the 0th neuron in the first layer. As a result, the other weights corresponding to input elements with 0 are not updated, which makes it difficult to learn the correlations between the input elements. It has been reported that applying soft label encoding (SLE) to nominal scales in auxiliary data improves accuracy [33]. In SLE, the correlation between features can be learned by replacing the elements that are 0 in one-hot encoding with 0.1. Actually, in the literature [33], SLE (Figure 6) enabled the learning of correlations within auxiliary data and enhanced the representational ability. Thus, for the auxiliary data used in this study, applying SLE to the discrete qualitative variables is expected to improve the classification accuracy. Consequently, in the proposed method, SLE is applied to the discrete values, and a vector combined with continuous values is input to the MLP to calculate the auxiliary feature ${\mathit{x}}_t^{\left(\mathrm{aux}\right)}\in {\mathbb{R}}^{d_{\mathrm{aux}}}$ at timestep t.

3.2. Feature Integration Based on Cross-Attention Mechanism

This section explains the cross-attention-based feature integration method. In the cross-attention module, the importance of each element in the features is determined using the query $\mathit{q}\in {\mathbb{R}}^{T\times d_m^{\prime }}$, key $\mathit{k}\in {\mathbb{R}}^{T\times d_m^{\prime }}$ and the value $\mathit{v}\in {\mathbb{R}}^{T\times d_m^{\prime }}$ ($m\in \{\mathrm{vis},\mathrm{aux}\}$, $d_m^{\prime }=d_m/h$). Here, h is a hyperparameter. The tuple ($\mathit{q}$, $\mathit{k}$, $\mathit{v}$) for each feature is calculated as follows:

$$\begin{array}{cc}\hfill {\mathit{q}}^m& ={\mathit{X}}^m{{\mathit{W}}^{(\mathrm{q},m)}}^{\top },\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathit{k}}^m& ={\mathit{X}}^m{{\mathit{W}}^{(\mathrm{k},m)}}^{\top },\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\mathit{v}}^m& ={\mathit{X}}^m{{\mathit{W}}^{(\mathrm{v},m)}}^{\top },\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.m\in \{\mathrm{vis},\mathrm{aux}\},\hfill \end{array}$$

(6)

where ${\mathit{W}}^{(\mathrm{q},m)}\in {\mathbb{R}}^{d_m^{\prime }\times d_m}$, ${\mathit{W}}^{(\mathrm{k},m)}\in {\mathbb{R}}^{d_m^{\prime }\times d_m}$ and ${\mathit{W}}^{(\mathrm{v},m)}\in {\mathbb{R}}^{d_m^{\prime }\times d_m}$ are the trainable parameters. In addition, ${\mathit{X}}^m=[{{\mathit{x}}_1^m}^{\top },{{\mathit{x}}_2^m}^{\top },\dots ,{{\mathit{x}}_T^m}^{\top }]\in {\mathbb{R}}^{T\times d_m}$. Next, using the tuple ($\mathit{q}$, $\mathit{k}$, $\mathit{v}$) among the heterogeneous features, the cross-attention $\mathrm{CA}(·,·,·)$ is calculated as follows:

$$\begin{array}{cc}\hfill \mathrm{CA}\left({{\mathit{q}}^m}^{\prime },{\mathit{k}}^m,{\mathit{v}}^m\right)=& \left[{\mathit{head}}_1^{(m^{\prime },m)},{\mathit{head}}_2^{(m^{\prime },m)},\dots ,{\mathit{head}}_h^{(m^{\prime },m)}\right]{\mathit{W}}^{(\mathrm{o},m)},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\mathit{head}}_i^{(m^{\prime },m)}=& \mathrm{Softmax}\left(\frac{{{\mathit{q}}^m}^{\prime }{{\mathit{k}}^m}^{\top }}{\sqrt{d_y^m}}\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.m^{\prime }& \ne m,i=1,2,\dots ,h,\hfill \end{array}$$

(9)

where ${\mathit{W}}^{(\mathrm{o},m)}\in {\mathbb{R}}^{h{d}_m^{\prime }\times d_{m^{\prime }}}$ is the trainable parameter. Finally, feature integration is performed by applying residual connections to each feature and the output values of the cross-attention mechanism as follows:

$$\begin{array}{cc}\hfill {\widehat{\mathit{X}}}^m=& {\mathit{X}}^m+\mathrm{CA}({\mathit{q}}^m,{\mathit{k}}^{m^{\prime }},{\mathit{v}}^{m^{\prime }}),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\widehat{\mathit{X}}}^{\mathrm{int}}=& [{\widehat{\mathit{X}}}^{\mathrm{vis}},{\widehat{\mathit{X}}}^{\mathrm{aux}}].\hfill \end{array}$$

(11)

In the proposed method, vectorization is performed by applying mean pooling to the integrated feature ${\widehat{\mathit{X}}}^{\mathrm{int}}$, which is then input to the MLP to output the final classification results. Thus, using cross-attention-based feature integration, the proposed method corrects features using heterogeneous data and processes time-series data across multiple timesteps. As a result, the proposed method improves the detection and prediction accuracy of winter road surface conditions.

4. Experiments

Experiments were conducted to verify the effectiveness of the proposed classification method based on MMTransformer. In the following, Section 4.1 describes the experimental dataset, Section 4.2 explains the experimental settings, and Section 4.3 presents the experimental results and a corresponding discussion.

4.1. Experimental Dataset

Here, we describe the dataset used in the experiments. The experiments utilized the winter road surface images and auxiliary data discussed in Section 4.1 to verify the effectiveness of the proposed method on real-world data. In addition, the seven categories (dry, wet, black sherbet, white sherbet, snow, compacted snow, and ice) were reorganized into three new categories, i.e., dry/wet, sherbet, and snow/compacted snow/ice, to detect and predict the winter road surface conditions from a practical perspective. The experiments were designed to confirm the effectiveness of using data across multiple timesteps to detect and predict winter road surface conditions. The classifications of road surface conditions were made for {0, 1, 3} hours later when inputting data at $T(=\{1,3,5\left\}\right)$ timesteps. Here, the data at one timestep were acquired at 20-min intervals. Please note that the input data were used on a per-timestep basis, and the teacher labels were used on an hourly basis. The number of samples for each road surface condition and the experimental settings are shown in

Table 2. Breakdown of experimental data used to immediately predict (detect) the road surface condition (0 h later).

	Number of Timesteps Used as Input
	1		3		5
Road Surface Condition	Training	Test	Training	Test	Training	Test
Dry/Wet	6000	35,771	6000	36,388	6000	36,460
Sherbet	5829	2474	5921	2505	5939	2506
Snow/Compacted snow/Ice	4614	418	4740	420	4747	420
Sum	16,443	38,663	16,661	39,313	16,686	39,386

,

Table 3. Breakdown of experimental data used to predict the road surface condition one hour later.

	Number of Timesteps Used as Input
	1		3		5
Road Surface Condition	Training	Test	Training	Test	Training	Test
Dry/Wet	6000	35,738	6000	36,370	6000	36,444
Sherbet	5838	2477	5935	2504	5948	2506
Snow/Compacted snow/Ice	4604	416	4730	420	4739	420
Sum	16,442	38,631	16,665	39,294	16,687	39,370

and

Table 4. Breakdown of experimental data used to predict the road surface condition three hours later.

	Number of Timesteps Used as Input
	1		3		5
Road Surface Condition	Training	Test	Training	Test	Training	Test
Dry/Wet	6000	35,703	6000	36,332	6000	36,407
Sherbet	5836	2475	5942	2504	5954	2506
Snow/Compacted snow/Ice	4604	416	4730	420	4741	420
Sum	16,428	38,593	16,667	39,256	16,695	39,333

. In the multi-timestep experimental settings, missing data were imputed using the average values from the data at other timesteps. In addition, data from 2017 and 2018 were used as the training data without distinction of the location, and data from 2019 were used as the test data. Also, note that the number of samples in each category varied significantly in the training data; thus, to suppress the reduction in classification accuracy due to the imbalanced number of samples, random extraction was performed such that the number of samples belonging to each category was approximately equal. As a result, the number of samples in the training data was smaller than that of test data through such an undersampling operation.

4.2. Experimental Settings

Here, we describe the experimental settings. The MLP used in the proposed method comprised three layers, and the feature dimensions of the images and auxiliary data were set to $d_{\mathrm{vis}}=16$ and $d_{\mathrm{aux}}=16$, respectively. For the Transformer encoder in the proposed method, we employed the ViT-B/16 model [24], which was pretrained on ImageNet [30]. For the loss function, cross-entropy loss was used, and for the optimization method, the Adam optimizer [34] with a learning rate of 0.001 was employed. During the training, the batch size was set to 8, and the number of epochs was set to 10. Moreover, we set $h=4$ as the hyperparameter.

To verify the effectiveness of the cross-attention–based feature integration implemented in the proposed method, we compared a method (Concatenation) that does not employ cross-attention by replacing Equation (11) with the following expression:

$$\begin{array}{cc}\hfill {\widehat{\mathit{X}}}^{\mathrm{int}}& =[{\mathit{X}}^{\mathrm{vis}},{\mathit{X}}^{\mathrm{aux}}].\hfill \end{array}$$

(12)

To evaluate the performance of the detection and prediction results, accuracy, macro precision, macro recall, and macro F1 metrics were considered, which are frequently used in the machine learning field for multiclass classification tasks. Each evaluation metric is calculated as follows:

• Accuracy

  $$\begin{array}{c}\hfill \mathrm{Accuracy}=\frac{{\sum }_{l=1}^L{\mathrm{TP}}_l}{{\sum }_{l=1}^L({\mathrm{TP}}_l+{\mathrm{FP}}_l)}.\end{array}$$

  (13)
• Macro Precision

  $$\begin{array}{cc}\hfill \mathrm{Macro}\mathrm{Precision}& =\frac{1}{L}\sum _{l=1}^L{\mathrm{Precision}}_l,\hfill \end{array}$$

  (14)

  $$\begin{array}{cc}\hfill {\mathrm{Precision}}_l& =\frac{{\mathrm{TP}}_l}{{\mathrm{TP}}_l+{\mathrm{FP}}_l}.\hfill \end{array}$$

  (15)
• Macro Recall

  $$\begin{array}{cc}\hfill \mathrm{Macro}\mathrm{Recall}& =\frac{1}{L}\sum _{l=1}^L{\mathrm{Recall}}_l,\hfill \end{array}$$

  (16)

  $$\begin{array}{cc}\hfill {\mathrm{Recall}}_l& =\frac{{\mathrm{TP}}_l}{{\mathrm{TP}}_l+{\mathrm{FN}}_l}.\hfill \end{array}$$

  (17)
• Macro F1

  $$\begin{array}{cc}\hfill \mathrm{Macro}\mathrm{F}1& =\frac{1}{L}\sum _{l=1}^L{\mathrm{F}1}_l,\hfill \end{array}$$

  (18)

  $$\begin{array}{cc}\hfill {\mathrm{F}1}_l& =\frac{2\times {\mathrm{Recall}}_l\times {\mathrm{Precision}}_l}{{\mathrm{Recall}}_l+{\mathrm{Precision}}_l}.\hfill \end{array}$$

  (19)

Here, ${\mathrm{TP}}_l$ and ${\mathrm{FN}}_l$ represent the number of true positive samples and false negative samples for the lth category, respectively, and ${\mathrm{FP}}_l$ denotes the number of false positive samples for the lth category.

4.3. Results and Discussion

4.3.1. Effectiveness of Time-Series Analysis

The experimental results obtained with different numbers of timesteps in the input data are shown in

Table 5. Experimental results obtained when varying the number of timesteps in the experiment to immediately predict (detect) the road surface condition.

Method	Accuracy	Macro Precision	Macro Recall	Macro F1
MMTransformer w/1 timestep	0.954	0.689	0.768	0.702
MMTransformer w/3 timesteps	0.958	0.710	0.791	0.735
MMTransformer w/5 timesteps	0.956	0.698	0.808	0.740

,

Table 6. Experimental results obtained when varying the number of timesteps in the experiment to predict the road surface condition one hour later.

Method	Accuracy	Macro Precision	Macro Recall	Macro F1
MMTransformer w/1 timestep	0.941	0.633	0.774	0.678
MMTransformer w/3 timesteps	0.944	0.636	0.791	0.683
MMTransformer w/5 timesteps	0.948	0.667	0.799	0.717

and

Table 7. Experimental results obtained when varying the number of timesteps in the experiment to predict the road surface condition three hours later.

Method	Accuracy	Macro Precision	Macro Recall	Macro F1
MMTransformer w/1 timestep	0.919	0.560	0.746	0.612
MMTransformer w/3 timesteps	0.926	0.579	0.747	0.627
MMTransformer w/5 timesteps	0.936	0.625	0.737	0.666

. Under all experimental conditions, the increase in the number of timesteps resulted in a higher macro F1 score, and we confirmed the effectiveness of using multiple timesteps when detecting and predicting the winter road surface conditions. On the other hand, when comparing MMTransformer w/5 with MMTransformer w/3 in Table 5, the macro Precision score decreased. Similarly, when comparing MMTransformer w/5 with MMTransformer w/3 in Table 7, the macro Recall score decreased. These score decreases were caused by differences in ${\mathrm{FP}}_l$ for macro Precision and in ${\mathrm{FN}}_l$ for macro Recall; however, both ${\mathrm{FP}}_l$ and ${\mathrm{FN}}_l$ should be evaluated for the classification model. Thus, we mainly focused on the harmonic mean of macro Precision and macro Recall, i.e., macro F1, and discussed the difference in the performance based on the macro F1. Thus, the effectiveness of time-series analysis with input data at multiple timesteps in the proposed method has been verified.

4.3.2. Effectiveness of Cross-Attention Mechanism

The experimental results comparing the proposed method with other methods are shown in

Table 8. Comparison of results in experiments to immediately predict (detect) road surface conditions.

Method	Accuracy	Macro Precision	Macro Recall	Macro F1
Concatenation w/5 timesteps	0.957	0.697	0.796	0.722
MMTransformer w/5 timesteps	0.956	0.698	0.808	0.740

,

Table 9. Comparison of results in experiments to predict road surface conditions one hour later.

Method	Accuracy	Macro Precision	Macro Recall	Macro F1
Concatenation w/5 timesteps	0.944	0.647	0.799	0.701
MMTransformer w/5 timesteps	0.948	0.667	0.799	0.717

and

Table 10. Comparison of results in experiments to predict road surface conditions three hours later.

Method	Accuracy	Macro Precision	Macro Recall	Macro F1
Concatenation w/5 timesteps	0.927	0.590	0.756	0.644
MMTransformer w/5 timesteps	0.936	0.625	0.737	0.666

. As can be seen, the macro F1 score of the proposed method surpasses that of the compared methods, which confirms the effectiveness of the MMTransformer. Specifically, by comparing MMTransformer and Concatenation, we verified that the cross-attention-based feature integration is effective for the classification of winter road surface conditions. On the other hand, when comparing MMTransformer w/5 with Concatenation w/5 in Table 10, the macro Recall score decreased. As well as macro Precision in Table 5 and macro Recall in Table 7, we mainly focused on the harmonic mean of macro Precision and macro Recall, i.e., macro F1, and discussed the difference in the performance based on the macro F1.

Thus, the effectiveness of using feature integration based on the cross-attention mechanism as the feature integration method has been verified. In addition, confusion matrices for the classification results of Concatenation w/5 timesteps and MMTransformer w/5 timesteps are shown in Figure 7, Figure 8 and Figure 9. In Figure 7 and Figure 8, the number of samples classified correctly for the dry/wet and snow/compacted snow/ice categories is approximately the same for both the MMTransformer and Concatenation. For the sherbet category, the MMTransformer outperformed the Concatenation considerably in terms of the number of correctly classified samples. In Figure 9, the number of correctly classified samples for the sherbet and snow/compacted snow/ice categories is similar; however, the MMTransformer outperformed the Concatenation considerably in the dry/wet category. These results confirm that the MMTransformer can predict winter road surface conditions more accurately than the Concatenation. However, when predicting the winter road surface conditions three hours later, as shown in Figure 9, there was no significant improvement in terms of classification accuracy for the important sherbet and snow/compacted snow/ice categories, which are critical for the effective detection and prediction of winter road surface conditions. Thus, improving the accuracy of predictions for winter road surface conditions at later times remains a challenge for future work.

4.3.3. Qualitative Evaluation through Visualization

In the MMTransformer, the output values obtained from the ViT model’s intermediate layers are used as image features. The ViT model employs an attention mechanism that recognizes important regions in images automatically and applies weighting to these regions. To achieve this, attention rollout [35], which presents the regions focused on by ViT through visualizing the weights in the attention mechanism, has been proposed. The regions presented by attention rollout are expected to serve as a basis for the rationale behind the classification results obtained by the ViT. In the proposed method, by observing the regions for winter road surface images, it is possible to gain insights into the relationship between the winter road surface images and winter road surface conditions and to use this information to enhance the performance of the classification model.

Figure 10 shows a visualization example obtained by applying attention rollout to the ViT encoder in MMTransformer, where redder regions are of higher interest in MMTransformer, and bluer regions are of lower interest. Here, the visualization was performed for MMTransformer w/5 timesteps in the experimental setting to detect the winter road surface conditions. As can be seen, there is more attention on the snow at the roadside at 20:00 and 20:40, and there is consistent attention to certain parts of the road surface over all timesteps. These observations imply that MMTransformer w/5 timesteps recognizes the presence of snow on the roadside but correctly identifies the road surface condition as sherbet due to the lesser amount of snow compared to the snow/compacted snow/ice conditions. From this result, it can be inferred that MMTransformer w/5 timesteps performs detection and prediction by focusing on the snow accumulation on the surface of the road in the images. Thus, by outputting the visualization results for the input images, we can gain insights into the relationship between the winter road surface images and the road surface conditions, and these insights can be used to enhance the performance of detection and prediction models.

5. Conclusions

This paper has proposed the MMTransformer method, which uses time-series data to detect and predict winter road surface conditions. The proposed method enhances the representational ability of the integrated features by performing feature correction through mutual complementation between modalities based on a cross-attention-based feature integration method for multiple modalities, e.g., road surface images and auxiliary data. In addition, by introducing time-series processing for the input data at multiple timesteps, the proposed method can integrate features in consideration of the temporal changes in winter road surface conditions. As a result, the proposed method improves the classification accuracy of winter road surface conditions by introducing a new integration for multiple modalities and time-series processing.

Experiments confirmed that the proposed MMTransformer method achieves high accuracy in classifying winter road surface conditions and is effective for both the detection and prediction tasks by varying the teacher labels. In addition, using attention rollout for visualization, we expected to provide additional insights into the relationship between road surface images and road surface conditions. In this way, as the experimental findings, it was implied that attention rollout works well for the multimodal classification model of winter road surface conditions. The visualization in the image encoder can be utilized to enhance the classification model when detecting and predicting road surface conditions, and the experimental findings discussed in this paper have demonstrated the potential of this technique.

On the other hand, confusion matrices indicate that performance improvement was slight for the data belonging to sherbet or snow/compacted snow/ice categories since the road surface images belonging to sherbet or snow/compacted snow/ice categories were visually similar to those of each other category. Such limitations caused by visual similarity can be solved by effectively leveraging non-visual information, including auxiliary data, which remains in future works.