Super-Resolution Reconstruction-Based Plant Image Classiﬁcation Using Thermal and Visible-Light Images

: Few studies have been conducted on thermal plant images. This is because of the difﬁculty in extracting and analyzing various color-related patterns and features from the plant image obtained using a thermal camera, which does not provide color information. In addition, the thermal camera is sensitive to the surrounding temperature and humidity. However, the thermal camera enables the extraction of invisible patterns in the plant by providing external and internal heat information. Therefore, this study proposed a novel plant classiﬁcation method based on both the thermal and visible-light plant images to exploit the strengths of both types of cameras. To the best of our knowledge, this study is the ﬁrst to perform super-resolution reconstruction using visible-light and thermal plant images. Furthermore, a method to improve the classiﬁcation performance through generative adversarial network (GAN)-based super-resolution reconstruction was proposed. Through the experiments using a self-collected dataset of thermal and visible-light images, our method shows higher accuracies than the state-of-the-art methods.


Introduction
There are many studies on plant classification based on images captured with visiblelight cameras.However, in order to obtain external and internal patterns and features of a plant that cannot be obtained with visible-light cameras, thermal cameras have recently begun to be used.However, studies on plant image classification using thermal cameras are scarce.A visible-light camera is sensitive to light and may produce low-quality images in a low-illumination environment or because of illumination change.Further, a thermal camera, which is sensitive to temperature and humidity, produces low-quality images owing to radiation emitted from or reflected by the various objects in the surroundings.Considering these challenges and strengths of the visible-light and thermal cameras, this study examined the use of both thermal and visible-light plant images.In addition, superresolution reconstruction (SRR) was performed to establish a classification method with improved performance compared to the previous studies.This is the first study to use visible-light and thermal images to perform SRR.The proposed method is described in detail in Section 3. Additionally, various SRR and classification experiments were conducted by using a self-collected thermal and visible-light plant image database.The novelty and innovations of this study are as follows: − Most previous studies focused on plant SRR based on visible-light images; however, no study exists on thermal-based images.This study examined SRR using thermal and visible-light plant images for the first time.In addition, this study proposed an SRRbased multiclass classification method using thermal and visible-light plant images.
− In this study, a novel method of a plant super-resolution (PlantSR) network was proposed with low-resolution (LR) thermal images (200 × 200 × 1) or LR visible-light images (200 × 200 × 3) as the input.In addition, a new residual blocks-in-residual block (RBRB) was employed in the structures of PlantSR and plant classification, thereby increasing the accuracy of plant classification.− In this study, a novel method of a plant multiclass classification (PlantMC) network was proposed using high-resolution (HR) thermal images and HR visible-light images as the input.To reduce the processing time of the PlantMC network, the input images, HR thermal images (600 × 600 × 1) and HR visible-light images (600 × 600 × 3) were cropped and converted into images with sizes of 200 × 200 × 25 and 200 × 200 × 75, respectively, through channel-wise concatenation.In addition, the accuracy of plant classification was improved using a new RBRB in the structure of PlantMC.− The PlantSR and PlantMC models proposed in the study and the self-collected thermal and visible-light plant image database were made available to other researchers [1].
The existing classification studies based on plant images can be categorized into plant image classification with SRR and plant image classification without SRR, where only visible-light images are used by the previous plant image-based SSR studies.Detailed explanations of the relevant works are presented in Section 2. Further, the experimental methods employed in this study are described in Section 3. The results and analysis of the experiments employing such methods are presented in Section 4. Finally, the discussion and conclusion are provided in Sections 5 and 6, respectively.

Related Works 2.1. Plant Image Classification without Super-Resolution Reconstruction
This subsection discusses the existing plant image classification studies without SRR.These studies were divided into three parts: visible-light image-based, thermal image-based and thermal-and visible-light image-based studies.

Visible-Light Image-Based Studies
Existing plant visible-light image-based classification studies are as follows.A study on crop disease classification [2] was conducted using the PlantDoc database and AAR network.Another method of crop disease classification [3] was performed using the Plant-Doc database and DenseNet-121 model.A different study on crop disease classification [4] employed the PlantDoc database and OMNCNN network to conduct the experimentation.A demonstration of crop and crop disease classification [5] utilized the PlantDoc database and five different deep learning methods (MobileNetV1, MobileNetV2, NASNetMobile, DenseNet121 and Xception).Another study on crop and crop disease classification [6] proposed a trilinear convolutional neural network model (T-CNN), while conducting various experiments using PlantVillage [7], PlantDoc [8] databases and pre-trained model with ImageNet.A prior study on plant image classification [9] was conducted using two models: periodic implicit generative adversarial networks (PI-GANs) and PI-CNN.The study employed four datasets (PlantVillage, PlantDoc, Fruits-360 and Plants) and performed various experiments while reducing the number of frames in the video.Augmentation was performed using PI-GAN and classification was conducted using PI-CNN.
Although these studies used visible-light cameras, the light-sensitive visible-light cameras have the disadvantage of producing low-quality images due to shadow, low illumination, illumination change and ambient light and its reflection.In addition, visiblelight cameras cannot be used to capture images at night without light.These limitations motivated the proposal of thermal plant image-based methods, which are explained in Section 2.1.2.

Thermal Image-Based Studies
A study on plant image and disease image classification [10] proposed the Plant-DXAI model, and various experiments were conducted using a paddy crop dataset and self-collected dataset.In this study, the CNN-16 network was used to perform plant classification, and class activation map and discriminator were additionally employed in the training phase to improve the performance of CNN-16 for disease classification.
These studies only used thermal cameras, which are sensitive to temperature and humidity and, therefore, produce low-quality images owing to radiation emitted and reflected by various objects in the environment.To address these challenges, thermal and visible-light plant image-based methods have been explored, as introduced in Section 2.1.3.

Thermal and Visible-Light Image-Based Studies
In a previous study on plant image classification [11], a classification method based on both thermal and visible-light images was proposed.According to the method, binary classification was performed via simultaneous use of thermal and visible-light images.In addition, the thermal image and stereo visible-light images were obtained for the study from three types of camera sensors: thermal and dual visible-light cameras.By integrating the three types of images, the accuracy of classifying them into healthy and diseased plant images increased.Further, binary classification (healthy or diseased) was performed by processing the features extracted with manual feature extraction methods through analysis of variance (ANOVA) [12] and support vector machine (SVM) [13].However, such a method increases the computation time and complexity of the system, owing to the simultaneous use of three types of cameras.Moreover, this method employs manual feature extraction methods and, thus, appropriate features cannot be extracted.In addition, only binary classification was performed using this method and various plant images were not recognized.A study on plant image classification [14] was conducted using nonaligned thermal and visible-light images.It proposed PlantCR and performed multiclass classification by considering visible-light and thermal images as the model input and integrating the two types of images inside the model.
However, the classification performance is limited by the LR images.To overcome this challenge, plant image classification methods based on plant image SSR were developed, as further explained in Section 2.2.

Plant Image Classification with Super-Resolution Reconstruction
In this section, existing plant image classification studies with SRR are explained.A previous study [15] demonstrated SRR and conducted plant disease classification using HR images produced with SRR.Each of the LR-, HR-and SRR-produced HR images are used to perform plant disease classification and the results were compared.In addition, the study compared the performance of the super-resolution convolutional neural network (SRCNN) with those of cubic, bicubic, lanczos and nearest neighbor (NN).Further, AlexNet was used for disease classification.Another study [16] demonstrated SRR and performed plant disease diagnosis using HR images produced with SRR.In the study, GAN [17] was used to achieve SRR, and 23 residual-in-residual dense blocks (RRDBs) were employed by the generator network.In addition, multiclass classification was executed using CNN Diag as the disease classification method.
However, these studies used visible-light cameras, and there are no studies on SRR using thermal plant images.In addition, there are very few multiclass classification studies that have used thermal images, including the studies introduced in Section 2. However, no study was conducted for both SRR and classification using thermal and visible-light images.Such a gap motivates this study to demonstrate multiclass classification using thermal and visible-light plant images and conduct SRR to further improve the accuracy.The proposed methods, PlantSR and PlantMC, are thoroughly explained in Section 3.
The aforementioned studies are summarized in Table 1 for comparison.

Overall Procedure of the Proposed Method
In this section, the method proposed by this study is explained in detail.A flowchart describing the method is provided in Figure 1.As is evident, the plant thermal and visible-light images are considered as input and categorized into twenty-eight classes by integrating the extracted features.To improve the classification performance, the LR input image of 200 × 200 dimensions was expanded into a 600 × 600 HR image through PlantSR.Subsequently, the HR image was used as input for PlantMC by cropping it into an image of 200 × 200 dimensions to reduce the processing time of classification.Detailed explanations of preprocessing are provided in Section 3.2.In addition, detailed descriptions of the proposed structure of PlantSR and PlantMC networks are presented in Sections 3.3 and 3.4, respectively, using tables and figures.In addition, the dimensions of input and output images and parameters used for the structures are explained.

Preprocessing
In this section, the preprocessing introduced in Figure 1, which involves cropping the HR image of 600 × 600 dimensions into a 200 × 200 × N image, is explained in detail.Examples of preprocessing thermal and visible-light images are shown in Figures 2 and 3, with Figure 2 focusing on further details.The preprocessing procedure entails cropping the input image into 200 × 200 dimensions by shifting it from left to right and top to bottom.Twenty-five cropped images (200 × 200 × 1) were then used to produce one 200 × 200 × 25 image (Figure 2) through channel-wise concatenation.In the shifting pro-cess, the image was cropped while overlapping at half the size (p = 100) of the output image (s = 200) to not overlook the image pattern.In the case of the visible-light image, the number of channels is 3, which becomes 75 after the preprocessing operation (Figure 3).The output images produced in Figures 2 and 3 are used as input to the proposed PlantMC.Simply, in the preprocessing stage, we reduce the size of input images to decrease the processing time.To achieve that, an input image is sliced into smaller images and the images are combined into a single image but with more channels, as shown in Figures 2 and 3.In other words, we reduce the spatial size but increase the number of channels in an image.We confirmed that the preprocessing operation decreases the processing time, as shown in Section 4.4.Moreover, we can also confirm that the preprocessing operation does not decrease the accuracy of the proposed method, as shown in experiments in Section 4.2.

Overall Procedure of the Proposed Method
In this section, the method proposed by this study is explained in detail.A flowchart describing the method is provided in Figure 1.As is evident, the plant thermal and visiblelight images are considered as input and categorized into twenty-eight classes by integrating the extracted features.To improve the classification performance, the LR input image of 200 × 200 dimensions was expanded into a 600 × 600 HR image through PlantSR.Subsequently, the HR image was used as input for PlantMC by cropping it into an image of 200 × 200 dimensions to reduce the processing time of classification.Detailed explanations of preprocessing are provided in Section 3.2.In addition, detailed descriptions of the proposed structure of PlantSR and PlantMC networks are presented in Sections 3.3 and 3.4, respectively, using tables and figures.In addition, the dimensions of input and output images and parameters used for the structures are explained.

Preprocessing
In this section, the preprocessing introduced in Figure 1 2) through channel-wise concatenation.In the shifting process, the image was cropped while overlapping at half the size (p = 100) of the output image (s = 200) to not overlook the image pattern.In the case of the visible-light image, the number of channels is 3, which becomes 75 after the preprocessing operation (Figure 3).The output images produced in Figures 2 and 3 are used as input to the proposed PlantMC.Simply, in the preprocessing stage, we reduce the size of input images to decrease the processing time.To achieve that, an input image is sliced into smaller images and the images are combined into a single image but with more channels, as shown in Figures 2 and 3.In other words, we reduce the spatial size but increase the number of channels in an image.We confirmed that the preprocessing operation decreases the processing time, as shown in Section 4.4.Moreover, we can also confirm that the preprocessing operation does not decrease the accuracy of the proposed method, as shown in experiments in Section 4.2.

Detailed Structure of the Proposed PlantSR Network
The dimensions of input and output images in the generator network are 200 × 200 × X and 600 × 600 × X, respectively, where X represents the number of channels, which is one for the thermal image and three for the visible-light image.Tables 2 and 3 present the descriptions of the generator and discriminator networks, respectively.As shown in Tables 2-7, input (input image), input layer, group layer (group), up-sampling layer (Up3), convolution layer (conv2d), activation layers (tanh and sigmoid), leaky rectified linear unit (LReLU), fully connected layer (FC), RBRB layer (RBRB), max pooling layer (max_pool), residual block (res_block), parametric rectified linear unit (prelu) and additional operation layer (add) were employed to construct the generator and discriminator networks.The output (class#) of the FC layer was 2 (real or fake).The "Times" column in Tables 4 and 5 represents the number of times by which the corresponding layer is repeated.Each layer in the column of parameters indicates the sum of parameters of only that layer.Up3 indicates triple up-sampling.The filter size and stride of the conv2d layers were (3 × 3) and (1 × 1), respectively, as presented in Tables 2-7.The padding of the conv2d layers was (0 × 0) in Table 5 and (1 × 1) in Tables 4 and 7. Here, "#" indicates "number of" in all contents.In the generator network (Table 2), the number of parameters based on the thermal and visible-light images was 6,166,081 and 6,169,539, respectively.However, in the discriminator network (Table 3), the number of parameters based on the thermal and visible-light images was 998,593 and 1,002,051, respectively.Tables 2 and 3 list the number of parameters when the visible-light image was used.The italic format in Figure 4 indicates the layer number of the discriminator network.

Detailed Structure of the Proposed PlantSR Network
The dimensions of input and output images in the generator network are 200 × 200 × X and 600 × 600 × X, respectively, where X represents the number of channels, which is one for the thermal image and three for the visible-light image.Tables 2 and 3 present the descriptions of the generator and discriminator networks, respectively.As shown in Tables 2-7, input (input image), input layer, group layer (group), up-sampling layer (Up3), convolution layer (conv2d), activation layers (tanh and sigmoid), leaky rectified linear unit (LReLU), fully connected layer (FC), RBRB layer (RBRB), max pooling layer (max_pool), residual block (res_block), parametric rectified linear unit (prelu) and additional operation layer (add) were employed to construct the generator and discriminator networks.The output (class#) of the FC layer was 2 (real or fake).The "Times" column in Tables 4 and 5 represents the number of times by which the corresponding layer is repeated.Each layer in the column of parameters indicates the sum of parameters of only that layer.Up3 indicates triple up-sampling.The filter size and stride of the conv2d layers were (3 × 3) and (1 × 1), respectively, as presented in Tables 2-7.The padding of the conv2d layers was (0 × 0) in Table 5 and (1 × 1) in Tables 4 and 7. Here, "#" indicates "number of" in all contents.In the generator network (Table 2), the number of parameters based on the thermal and visible-light images was 6,166,081 and 6,169,539, respectively.However, in the discriminator network (Table 3), the number of parameters based on the thermal and visible-light images was 998,593 and 1,002,051, respectively.Tables 2 and 3 list the number of parameters when the visible-light image was used.The italic format in Figure 4 indicates the layer number of the discriminator network.

Detailed Structure of the Proposed PlantMC Network
The structure of the proposed PlantMC network is provided in Table 8 and Figure 5.The dimensions of the input and output images of the PlantMC network were 200 × × X and 28 × 1, respectively, where X represents the number of channels and is 25 for thermal images and 75 for visible-light images, respectively.The configuration of the group used

Detailed Structure of the Proposed PlantMC Network
The structure of the proposed PlantMC network is provided in Table 8 and Figure 5.The dimensions of the input and output images of the PlantMC network were 200 × 200 × X and 28 × 1, respectively, where X represents the number of channels and is 25 for thermal images and 75 for visible-light images, respectively.The configuration of the group used in Table 8 is identical to that of the discriminator network group shown in Table 5, with the exception of the stride size of the second conv2d being (2 × 2).The input layers (input layer_1 and 2), concatenate layer (concat) and activation layer (softmax) were used to describe the PlantMC network, as provided in Table 8, in contrast to Tables 2 and 3.The output of the FC layer (class#) is 28.The parameter# column in Table 8 presents the two types of parameter numbers in distinct formats and compares them.For example, the parameter number obtained using the thermal and visible images with dimensions of 600 × 600 × 1 and 600 × 600 × 3, respectively, produced by PlantSR, is provided in non-italic format.However, the parameter number when the preprocessed thermal and visible-light images with dimensions of 200 × 200 × 25 and 200 × 200 × 75, respectively, is shown in italic format for comparison.As shown in Table 8, the number of parameters was lower by 159,744 when using images obtained using preprocessing.The remaining values in Table 8 were consistent with the descriptions in Section 3.3.

Details of Database and Experimental Setup
We focused on using both thermal and visible-light images to increase the accuracy of plant classification in this study.However, the existing open datasets, such as PlantVillage [7] and PlantDoc [8] or the others, do not provide thermal images.Therefore, we did not use the datasets in our experiments.Therefore, in this study, the experiments were conducted using the TherVisDb [18] dataset comprising images of various roses and rose leaves.The dataset was obtained in July 2022.The details of the dataset are described in three different tables.In detail, Table 9 describes the names of flowers and class indices; the number of images of each class and three subsets for the 3-fold cross-validation; the number of images in the validation set; the number of thermal and visible-light images, separately; and the total number of images in the dataset.In the table, the 'Image#' column represents the sum of numbers of thermal and visible-light images.The 'Sets 1-3' column indicates the dataset split for 3-fold cross-validation.Table 10 describes the weather information of the day when we collected our dataset, including humidity, temperature, wind speed, fine dust, ultra-fine dust and UV index.Moreover, Table 11 describes other information, such as image dimension, depth and extension, before and after augmentation.Moreover, the number of total classes and information of camera sensors were described.In addition, examples of thermal images and corresponding visible-light images in the dataset are presented in Figure 6.

Details of Database and Experimental Setup
We focused on using both thermal and visible-light images to increase the accuracy of plant classification in this study.However, the existing open datasets, such as PlantVillage [7] and PlantDoc [8] or the others, do not provide thermal images.Therefore, we did not use the datasets in our experiments.Therefore, in this study, the experiments were conducted using the TherVisDb [18] dataset comprising images of various roses and rose leaves.The dataset was obtained in July 2022.The details of the dataset are described in three different tables.In detail, Table 9 describes the names of flowers and class indices; the number of images of each class and three subsets for the 3-fold cross-validation; the number of images in the validation set; the number of thermal and visible-light images, separately; and the total number of images in the dataset.In the table, the 'Image#' column represents the sum of numbers of thermal and visible-light images.The 'Sets 1-3' column indicates the dataset split for 3-fold cross-validation.Table 10 describes the weather information of the day when we collected our dataset, including humidity, temperature, wind speed, fine dust, ultra-fine dust and UV index.Moreover, Table 11 describes other information, such as image dimension, depth and extension, before and after augmentation.Moreover, the number of total classes and information of camera sensors were described.In addition, examples of thermal images and corresponding visible-light images in the dataset are presented in Figure 6. (a)

Details of Database and Experimental Setup
We focused on using both thermal and visible-light images to increase the accuracy of plant classification in this study.However, the existing open datasets, such as PlantVillage [7] and PlantDoc [8] or the others, do not provide thermal images.Therefore, we did not use the datasets in our experiments.Therefore, in this study, the experiments were conducted using the TherVisDb [18] dataset comprising images of various roses and rose leaves.The dataset was obtained in July 2022.The details of the dataset are described in three different tables.In detail, Table 9 describes the names of flowers and class indices; the number of images of each class and three subsets for the 3-fold cross-validation; the number of images in the validation set; the number of thermal and visible-light images, separately; and the total number of images in the dataset.In the table, the 'Image#' column represents the sum of numbers of thermal and visible-light images.The 'Sets 1-3' column indicates the dataset split for 3-fold cross-validation.Table 10 describes the weather information of the day when we collected our dataset, including humidity, temperature, wind speed, fine dust, ultra-fine dust and UV index.Moreover, Table 11 describes other information, such as image dimension, depth and extension, before and after augmentation.Moreover, the number of total classes and information of camera sensors were described.In addition, examples of thermal images and corresponding visible-light images in the dataset are presented in Figure 6.In the "Before augmentation" part in Table 11, a single image included many plants; therefore, such images were cropped into images with a size of 300 × 300 to increase the number of images detailed in the "After augmentation" part in Table 11.The number of images produced as such for each class in the dataset is provided in Table 9.In the training phase, each training set was expanded using augmentation methods (rotating three times by 90 • and flipping horizontally).In addition, down-sizing of images from dimensions of 300 × 300 to 200 × 200 was performed to conduct SRR.The computer hardware and software used in this study are described in Table 12.

Experimental Results
This section is divided into four parts: training setup, testing, experimental comparison and processing time.In Section 4.1, the setup of the training phase, characterized by, for example, the hyperparameters and training loss, is described.In Section 4.2, the results obtained from the training phase are presented.Thereafter, the experimental results produced by the existing and proposed methods are compared in Section 4.3.Finally, the processing time of the proposed method is measured in Section 4.4.

Training Setup
The training setup of the proposed methods is presented in Table 13.The training loss and validation accuracy curves of PlantSR and PlantMC are shown in Figure 7. Figure 7a,b show the training and validation loss curves of the PlantSR per epoch.Figure 7c,d show the loss and accuracy curves of the PlantMC per epoch.As is evident, the network developed in this study was sufficiently trained without being overfitted by the training data.In addition, Table 13 presents search spaces and selected values of the hyperparameters for the network.

Testing
The testing results are presented in this section.To measure the accuracy of PlantSR, Equation ( 1) for peak signal-to-noise ratio (PSNR) [28] and Equation ( 2) for structural similarity index measure (SSIM) [29] were used.Further, Equations ( 3)-( 6) were employed to calculate the accuracy of PlantMC.
where X, Y, M and N represent original image, restored image, image width and image height, respectively.
where µ X and σ

Testing
The testing results are presented in this section.To measure the accuracy of PlantSR, Equation ( 1) for peak signal-to-noise ratio (PSNR) [28] and Equation ( 2) for structural similarity index measure (SSIM) [29] were used.Further, Equations ( 3)-( 6) were employed to calculate the accuracy of PlantMC.
where X, Y, M and N represent original image, restored image, image width and image height, respectively.
Tables 14-16 present the results of performing classification using the image reconstructed with PlantSR as the input for PlantMC.Table 14 presents and compares each of the results of 3-fold cross-validation.As shown in Table 14, accuracies obtained in the three folds are almost similar to each other and fold-1 showed the highest accuracy, whereas fold-3 showed the lowest.Table 15 compares the results from the method using RBRB (two residual blocks followed by an addition operation, as shown in Table 6) and Tables 14-16 present the results of performing classification using the image reconstructed with PlantSR as the input for PlantMC.Table 14 presents and compares each of the results of 3-fold cross-validation.As shown in Table 14, accuracies obtained in the three folds are almost similar to each other and fold-1 showed the highest accuracy, whereas fold-3 showed the lowest.Table 15 compares the results from the method using RBRB (two residual blocks followed by an addition operation, as shown in Table 6) and those from the method not using RBRB (two residual blocks not followed by an addition operation).Evidently, the accuracy results produced using RBRB were greater.Therefore, we used the RBRB in further experiments in this study; moreover, we propose the RBRB in this study for the increased accuracy of the plant image classification.The results using images modified through preprocessing, of 200 × 200 × 25 (75) dimensions, those using images without preprocessing, of 600 × 600 × 1 (3) dimensions, are provided for comparison in Table 16.Although preprocessing did not improve the accuracy by a significant amount, it decreased the number of parameters and reduced the memory size, as demonstrated in Table 8.In addition, it reduced the processing time, as is evident in Section 4.4.Table 17 presents the results of using thermal images as the input to PlantSR (PlantSR (Th)) and results of using visible-light images as the input to PlantSR (PlantSR (V)).In addition, PlantMC results, with and without PlantSR, are compared.Further, the accuracy of each proposed method is provided with class-based comparisons.As demonstrated, the accuracy of classification with SRR (PlantSR + PlantMC) was higher than that of classification without SRR (PlantMC).Moreover, Spiraea salicifolia l and White symphonie showed the highest PSNRs of 27.44 and 28.33 in PlantSR (Th) and PlantSR (V), respectively, whereas Oklahoma and Kerria japonica showed the lowest PSNRS of 26.47 and 27.38 in PlantSR (Th) and PlantSR (V), respectively.Moreover, Kerria japonica showed the highest F1-scores of 99.63 and 100 in both PlantMC and PlantSR+PlantMC, respectively, whereas Cleopatra showed the lowest F1-scores of 71.62 and 72.37 in both PlantMC and PlantSR + PlantMC, respectively.Figures 8 and 9 show that the images produced using the proposed method exhibited sharper contrast than those produced using the bicubic method, which means that the proposed method can generate plant images with higher contrast than the conventional bicubic method.As shown in Table 18, the accuracy and classification efficiency results obtained using PlantSR were higher than those obtained using bicubic.Table 18 also demonstrates that classification performance using bicubic and PlantSR-reconstructed images exhibited higher efficiency compared to the results using the original image.This analysis supports the positive influence of SRR on the classification performance.Moreover, it means that the proposed SRR method increases the performance of the classification method.

Comparisons with the Existing Methods
In this section, the proposed methods are compared to the state-of-the-art methods experimentally.As this study is the first to perform SRR and classification using plant thermal and visible-light images, there are no similar existing methods for a fair comparison, as supported by Table 1.Therefore, the existing methods of SRR and classification using plant visible-light images [15,16] were used for comparative analysis.The existing plant image SRR and classification methods [15,16] were compared to the proposed methods (PlantSR, PlantMC and PlantSR + PlantMC), as presented in Tables 19-21.When the experimental comparison was conducted, training was executed on the TherVisDb database for the existing methods [15,16].The detailed explanations on the SRR and CNN configuration used by the existing method-1 [15] can be found in [15] and [33], respectively.Further, the detailed description of the SRR and CNN configuration used by the existing method-2 can be found in [16,34].The previous study [16] used a modified VGG-Net proposed in study [34] as a classification network.The experimental results for plant image SRR and plant image classification were compared in Sections 4.3.1 and 4.3.2,respectively.In this section, the existing plant image-based SRR methods are used for comparative analysis.As is evident in Table 1, there is no existing study on plant thermal imagebased SRR method; thus, the plant visible-light image was used for comparison.The SRR methods employed in Method-1 and Method-2 were used in comparison in Table 19, and the accuracy results obtained from the proposed method were the highest.Moreover, the proposed method showed the highest accuracy, whereas the Bicubic showed the lowest accuracy.In addition, the SRR methods showed very little difference in accuracy.Figure 10 presents a graphical comparison of the SRR methods.

Comparisons with Plant Image Classification Methods
In this section, the existing plant image-based SRR and classification methods were used for comparative analysis.While only visible-light images were used by Method-1 and Method-2, both thermal and visible-light images were used by the proposed method.Under both experimental conditions, the proposed method produced the highest accurate results, as is evident in Tables 20 and 21.These results confirmed that the use of thermal and visible-light images simultaneously, as in the proposed method, was efficient.In addition, Tables 18, 20 and 21 demonstrate that classification results using SRR are more accurate, thereby confirming that SRR is effective in plant classification.Moreover, in all cases, the proposed method showed higher accuracies than the other methods.In Tables 19-21, the outperformance of the proposed method is not significant.Because the previous networks (method-1 [15] and method-2 [16]) used larger numbers of parameters, layers and additional networks for super-resolution reconstruction (SRR), the proposed method could not outperform their results significantly.For example, a classification network of method-1 [15] has over 62 million parameters, whereas our classification network has only 5 million parameters.Moreover, they used a super-resolution convolutional neural network (SRCNN) for SRR, which was pretrained with the ImageNet database, including huge numbers of images.However, the SRCNN was not used in our method, which makes it difficult to achieve a significant result compared to the previous method [15].The classification networks of method-2 [16] and the proposed method are similar; however, their SRR network has a total of 115 convolution layers, whereas our SRR network has only 12 convolution layers.Accordingly, it is difficult to achieve a significant result compared to the previous method [16].

Processing Time of the Methods
The processing time of the proposed PlantSR and PlantMC in the testing phase is shown in Table 22.Table 22 presents the frame rates of PlantSR methods using thermal and visible-light images, which were 14.6 and 15.93 frames per second (fps), respectively.The frame rate of the PlantSRs + PlantMC method was 5.63 fps.In addition, the frame rate when PlantSR-produced thermal and visible-light images with dimensions of 600 × 600 × 1 and 600 × 600 × 3, respectively, were used was 17.17 fps.As demonstrated, the frame rate obtained according to the method proposed in Figures 2 and 3

Discussion
In this study, SRR and classification using plant thermal and visible-light images were examined.Using the two types of images as separate inputs for the proposed method (PlantSR) produced bigger images with sharper contrast, as demonstrated in Figures 8 and 9. Further, the proposed method (PlantMC) exhibited higher improved performance when PlantSR was employed to produce and integrate the two types of images, as shown in Table 17.As demonstrated by Tables 20 and 21, the proposed method delivered higher performance than the methods introduced by the existing studies.In addition, the method of using the preprocessed thermal and visible-light images with dimensions of 200 × 200 × 25 (75) was faster than the method using the PlantSR-produced images with dimensions of 600 × 600 × 1 (3).The measures of accuracy of the two methods did not differ significantly, as shown in Table 16.Furthermore, the results of the method with RBRB (two residual blocks followed by an addition operation, as shown in Table 6) that is newly developed in this study were compared to those of the method without RBRB (two residual blocks not followed by an addition operation) in Table 15.Evidently, the accuracy results with the application of RBRB were higher.
Figures 11 and 12 show error cases by the proposed method, PlantMC.The classification error of PlantMC was influenced by the image that is wrongly reconstructed by PlantSR.As is evident, when the image was too bright or too dark, even brighter or darker images were produced.In contrast, the images produced using the bicubic appeared clearer than those produced by PlantSR.

Discussion
In this study, SRR and classification using plant thermal and visible-light images were examined.Using the two types of images as separate inputs for the proposed method (PlantSR) produced bigger images with sharper contrast, as demonstrated in Figures 8  and 9. Further, the proposed method (PlantMC) exhibited higher improved performance when PlantSR was employed to produce and integrate the two types of images, as shown in Table 17.As demonstrated by Tables 20 and 21, the proposed method delivered higher performance than the methods introduced by the existing studies.In addition, the method of using the preprocessed thermal and visible-light images with dimensions of 200 × 200 × 25(75) was faster than the method using the PlantSR-produced images with dimensions of 600 × 600 × 1(3).The measures of accuracy of the two methods did not differ significantly, as shown in Table 16.Furthermore, the results of the method with RBRB (two residual blocks followed by an addition operation, as shown in Table 6) that is newly developed in this study were compared to those of the method without RBRB (two residual blocks not followed by an addition operation) in Table 15.Evidently, the accuracy results with the application of RBRB were higher.
Figures 11 and 12 show error cases by the proposed method, PlantMC.The classification error of PlantMC was influenced by the image that is wrongly reconstructed by PlantSR.As is evident, when the image was too bright or too dark, even brighter or darker images were produced.In contrast, the images produced using the bicubic appeared clearer than those produced by PlantSR.

Conclusions
In this study, the SRR and plant classification methods based on thermal and light images were proposed and tested.The TherVisDb dataset was utilized in v experiments in the study.The dataset contained various images of roses and rose Based on the plant image classification experiment using TherVisDb, the accuracy proposed method appeared higher than that of existing methods, with an accu 99.22% and F1-score of 90.16%.In addition, the results of the plant image SRR expe using TherVisDb revealed that the proposed method was more accurate than the e method, as is evident from the PSNR of 27.84 and SSIM of 0.91.
The experimental results provided in Table 18 confirmed that the accuracy o fication increased when the images were expanded using PlantSR.As explained preprocessing in Section 3.2, by reducing the dimensions of PlantSR-produced imag increasing the number of channels in exchange, the processing time, number of p ters and memory size of PlantMC were decreased, as is evident in Tables 8 and 22 Further, Table 15 indicates that the use of RBRB newly developed in this stu confirmed to produce higher accuracy than the method using the basic residual bl

Conclusions
In this study, the SRR and plant classification methods based on thermal and visiblelight images were proposed and tested.The TherVisDb dataset was utilized in various experiments in the study.The dataset contained various images of roses and rose leaves.Based on the plant image classification experiment using TherVisDb, the accuracy of the proposed method appeared higher than that of existing methods, with an accuracy of 99.22% and F1-score of 90.16%.In addition, the results of the plant image SRR experiment using TherVisDb revealed that the proposed method was more accurate than the existing method, as is evident from the PSNR of 27.84 and SSIM of 0.91.
The experimental results provided in Table 18 confirmed that the accuracy of classification increased when the images were expanded using PlantSR.As explained for the preprocessing in Section 3.2, by reducing the dimensions of PlantSR-produced images and increasing the number of channels in exchange, the processing time, number of parameters and memory size of PlantMC were decreased, as is evident in Tables 8 and 22.
Further, Table 15 indicates that the use of RBRB newly developed in this study was confirmed to produce higher accuracy than the method using the basic residual block.

Figure 1 .
Figure 1.Overview of the proposed method for plant image SRR and classification.
, which involves cropping the HR image of 600 × 600 dimensions into a 200 × 200 × N image, is explained in detail.Examples of preprocessing thermal and visible-light images are shown in Figures 2 and 3, with Figure 2 focusing on further details.The preprocessing procedure entails cropping the input image into 200 × 200 dimensions by shifting it from left to right and top to bottom.Twenty-five cropped images (200 × 200 × 1) were then used to produce one 200 × 200 × 25 image (Figure

Figure 1 .Figure 2 .
Figure 1.Overview of the proposed method for plant image SRR and classification.

Figure 2 .
Figure 2. Example of preprocessing on thermal image.

Figure 2 .
Figure 2. Example of preprocessing on thermal image.

Figure 6 .
Figure 6.Example images of TherVisDb.From left to right: images of blue river, charm of Paris, Cleopatra and cocktail.(a) Visible-light images and (b) corresponding thermal images.

Figure 6 .
Figure 6.Example images of TherVisDb.From left to right: images of blue river, charm of Paris, Cleopatra and cocktail.(a) Visible-light images and (b) corresponding thermal images.

Figure 7 .
Figure 7. Loss and accuracy curves of the PlantSR and PlantMC.(a) Training loss curves of PlantSR; (b) validation loss curves of PlantSR; (c) training and validation loss curves of PlantMC; (d) training and validation accuracy curves of PlantMC.

Figure 8 .
Figure 8. Example thermal images generated by PlantSR.From top to bottom, i gaujard, Duftrausch and Elvis.(a) Original image; (b) enlarged image by bicubic [3 image by PlantSR.

Figure 9 .
Figure 9. Example visible-light images generated by PlantSR.From top to bottom, im symphonie, Roseraie du chatelet and Twist.(a) Original image; (b) enlarged image enlarged image by PlantSR.

Figure 9 .
Figure 9. Example visible-light images generated by PlantSR.From top to bottom, images of White symphonie, Roseraie du chatelet and Twist.(a) Original image; (b) enlarged image by bicubic; (c) enlarged image by PlantSR.
presents a graphical comparison of the SRR methods.

Figure 10 . 1 Figure 10 .
Figure 10.Example visible-light images generated by SRR methods.From top to bottom, images of White symphonie, Roseraie du chatelet and Twist.(a) Original image; (b) enlarged image by bicubic; (c) enlarged image by Method-1; (d) enlarged image by Method-2; (e) enlarged image by PlantSR.4.3.2.Comparisons with Plant Image Classification Methods In this section, the existing plant image-based SRR and classification methods were used for comparative analysis.While only visible-light images were used by Method-1 Figure 10.Example visible-light images generated by SRR methods.From top to bottom, images of White symphonie, Roseraie du chatelet and Twist.(a) Original image; (b) enlarged image by bicubic; (c) enlarged image by Method-1; (d) enlarged image by Method-2; (e) enlarged image by PlantSR.

Figure 11 .
Figure 11.Example of error cases of thermal images generated by PlantSR.From top to bottom, images of Rose gaujard, Duftrausch and Elvis.(a) Original image; (b) enlarged image by bicubic; (c) enlarged image by PlantSR.

Figure 11 .Figure 12 .
Figure 11.Example of error cases of thermal images generated by PlantSR.From top to images of Rose gaujard, Duftrausch and Elvis.(a) Original image; (b) enlarged image by bic enlarged image by PlantSR.

Figure 12 .
Figure 12.Example of error cases of visible-light images generated by PlantSR.From top to bottom, images of White symphonie, Roseraie du chatelet and Twist.(a) Original image; (b) enlarged image by bicubic; (c) enlarged image by PlantSR.

Table 1 .
Summary of existing classification and SRR studies on plant image databases.

Table 2 .
Description of the generator network of PlantSR.

Table 2 .
Description of the generator network of PlantSR.

Table 3 .
Description of the discriminator network of PlantSR.

Table 4 .
Group layer of generator network.

Table 5 .
Group layer of discriminator network.

Table 7 .
Description of a residual block.
Mathematics 2023, 11, x FOR PEER REVIEW of 23

Table 8 .
Description of the proposed PlantMC.

Table 9 .
Description of classes and dataset split.

Table 10 .
Weather information for the surrounding environment at the time of image acquisition.

Table 11 .
Other relevant information in the dataset.

Table 9 .
Description of classes and dataset split.

Table 12 .
Description of hardware and software of desktop computer.

Table 13 .
Search spaces and selected values of hyperparameters for the proposed methods.

Table 14 .
Comparison of accuracies obtained in different folds using PlantSR + PlantMC.

Table 15 .
Comparison of accuracies obtained by using classification methods with and without RBRB.

Table 16 .
Comparison of accuracies obtained using classification methods using images with different sizes and channels.

Table 17 .
Detailed accuracy of each class by the proposed PlantSR and PlantMC with and without PlantSR.

Table 18 .
Comparison of accuracies by bicubic and the proposed PlantSR, and PlantMC with bicubic and PlantSR.

Table 18 .
Comparison of accuracies by bicubic and the proposed PlantSR, and PlantM and PlantSR.

Table 19 .
Comparison of accuracies obtained by using SRR methods.

Table 20 .
Comparison of accuracies obtained by using classification methods without SRR.

Table 21 .
Comparison of accuracies obtained by using classification methods with SRR.

Table 19 .
Comparison of accuracies obtained by using SRR methods.

Table 22 .
Processing time of the methods per image (unit: ms).