Smartphone Application-Enabled Apple Fruit Surface Temperature Monitoring Tool for In-Field and Real-Time Sunburn Susceptibility Prediction

Heat stress and resulting sunburn is a major abiotic stress in perineal specialty crops. For example, such stress to the maturing fruits on apple tree canopies can cause several physiological disorders that result in considerable crop losses and reduced marketability of the produce. Thus, there is a critical technological need to effectively monitor the abiotic stress under field conditions for timely actuation of remedial measures. Fruit surface temperature (FST) is one of the stress indicators that can reliably be used to predict apple fruit sunburn susceptibility. This study was therefore focused on development and in-field testing of a mobile FST monitoring tool that can be used for real-time crop stress monitoring. The tool integrates a smartphone connected thermal-Red-Green-Blue (RGB) imaging sensor and a custom developed application (‘AppSense 1.0’) for apple fruit sunburn prediction. This tool is configured to acquire and analyze imagery data onboard the smartphone to estimate FST. The tool also utilizes geolocation-specific weather data to estimate weather-based FST using an energy balance modeling approach. The ‘AppSense 1.0’ application, developed to work in the Android operating system, allows visual display, annotation and real-time sharing of the imagery, weather data and pertinent FST estimates. The developed tool was evaluated in orchard conditions during the 2019 crop production season on the Gala, Fuji, Red delicious and Honeycrisp apple cultivars. Overall, results showed no significant difference (t110 = 0.51, p = 0.6) between the mobile FST monitoring tool outputs, and ground truth FST data collected using a thermal probe which had accuracy of ±0.4 °C. Upon further refinements, such tool could aid growers in real-time apple fruit sunburn susceptibility prediction and assist in more effective actuation of apple fruit sunburn preventative measures. This tool also has the potential to be customized for in-field monitoring of the heat stressors in some of the sun-exposed perennial and annual specialty crops at produce maturation.


Introduction
Sunburn is a major physiological disorder that adversely affects productivity of fruits and vegetables including apple, grapes, pepper, pumpkin and watermelons [1][2][3][4][5]. Fruit sunburn in apples has been reported in all major apple growing regions around the globe and causes 10-40% yield losses. Sunburn is caused by sunlight with high intensity, ultra-violet (UV) radiation and photo-synthetically active radiation (PAR) that elevates maturing fruit peel temperature during the field production. Sunburn necrosis, sunburn browning and photo-oxidative sunburn are the three major sunburn types [4,5]. Sunburn necrosis leads to thermal death of epidermal and sub-epidermal cells (peel) and takes place when fruit surface temperature (FST), a reliable indicator of sunburn, reaches 52 ± 1 • C. Sunburn browning occurs at a slightly lower FST, i.e., between 46 and 49 • C, whereas the photo oxidative sunburn occurs at a much lower FST because of PAR and light shocks to sudden sun-exposure of shaded apple due to management practices such as hand thinning, selective picking or pruning operations [2][3][4][5][6][7][8][9]. Apart from sunburn, FST also influences several physiological and biological processes in fruit such as flavor and nutritional values, sugar and acid content, ripening process, fruit quality, size, appearance, and insect/pest infestation susceptibility.
Fruit sunburn forecasting has been conventionally done using the air temperature data collected by open field weather stations. Often, such an approach is highly unreliable and can result in untimely application of water for evaporative cooling of canopies and fruits. Excess evaporative cooling can be water and energy inefficient and may create food safety issues. A weather-data-based apple surface-temperature dynamics model [10] that uses parameters such as wind speed, humidity, solar intensity, and dew point temperature, apart from air temperature, could be useful for FST estimates. However, it requires on-site sensors to get reliable micro-climate weather data. Thermocouples, thermometers and thermal probes are also used for FST estimation. However, such probes installation tends to be laborious, less accurate, data limited and destructive; limiting wide scale adaptation.
In recent years, portable miniaturized thermal imaging sensors have been explored in various crop-stress monitoring applications including that of rapid apple FST monitoring [10]. During field use, thermal imagery can be inadequate as it can have challenges in segmenting regions of interests (fruits) within the background branches, stems and leaves [11]. However, when combined with RGB imagery data, thermal imagery can be used for apple fruit surface temperature monitoring [11]. Our team has developed and successfully tested such smart apple sunburn sensing systems [12]. Nevertheless, these orchard sensing system needs to be rugged, economically viable, and connected to a reliable cellular network that may not be available at many orchards. These challenges with existing systems raise a need for a user friendly, low cost, and portable sunburn prediction tool that will aid growers in real-time, accurate, apple sunburn monitoring and pertinent decision-making.
Our team has envisioned such a mobile FST monitoring tool that leverages advances in low cost commercial thermal-RGB imaging sensors and smartphone technologies that allow ease of integration. Smartphones are becoming commonly available to wider demographics and have opened a new horizon for smartphone-based applications that helps in solving daily-life problems. Recently, researchers have taken advantage of the increased computing potential, accessibility, ease-of-operation and low cost of smartphones for developing pertinent use cases in medical diagnosis, environmental monitoring and food safety [13]. Related to agriculture applications, Wang et al. [14] utilized the smartphone built-in camera and developed an application for in-field fruit size estimation. Das et al. [15] has developed a prototype for a smartphone-based spectrometer for rapid and non-destructive fruit ripeness testing.
Overall, smartphone-based applications integrated with compatible imaging and optical sensors have a huge potential for effective orchard crop stress monitoring. Therefore, this study focused on the development of a smartphone-enabled, thermal-RGB, and microclimate-sensing assisted, real-time apple sunburn prediction tool. The specific study objectives were to: (1) Develop a smartphone enabled miniaturized and portable apple fruit surface temperature monitoring tool with a compatible application ('AppSense 1.0') for real-time fruit surface temperature estimation. (2) Evaluate the performance of the developed tool in orchard conditions towards predicting sunburn susceptibility of four fresh market apple cultivars cv. 'Gala', 'Fuji', 'Red delicious' and 'Honeycrisp'.

Mobile Fruit Surface Temperature Monitoring Tool
The mobile FST monitoring tool ( Figure 1) consists of a thermal-RGB sensor (FLIR One Pro, FLIR Systems, Inc., Wilsonville, Oregon, OR, USA) integrated with an Android operating system (OS) based smartphone. Typical design consideration was that such a tool should work on any smartphone that has on-the-go universal serial bus [OTG USB] and 'Android 8.0' or higher OS version. This tool uses an Android OS compatible application (version: 1.0; hereafter termed as 'AppSense 1.0') that allows data acquisition, processing, management and sharing with the end-users. The mobile FST tool enables real-time acquisition of in-field thermal-RGB images and geolocalized weather data for resulting data analysis on-board the smartphone to estimate apple FST. The imagery and weather-data-based FST estimates can be stored on board the smartphone and shared with end-users to make appropriate sunburn management related decisions. This tool was designed in such a way that a typical fruit grower or farm crew can install and use it with ease.

Smartphone Application Development
The 'AppSense 1.0' application was developed in 'Android Studio' integrated development environment (IDE) (open source, version 3.4.2). This application was designed to collect weather data from the nearest open field weather station, acquire the thermal-RGB image, estimate the weather and imagery based FSTs, save and display the processed images and FST data, and export them in excel format ('*.xlxs') ( Figure 2). An open source computer vision (OpenCV, version 3.4.0) Java library was utilized for on-board image processing and FST estimation. The 'AppSense 1.0'-captured raw and processed images were stored in the internal memory of the smartphone that can be utilized for further analysis. The data analyzed by the tool is stored in the local 'SQLite' database and can be retrieved and managed at a later stage.

Weather Data Based FST Estimation
An apple surface temperature dynamics model by Li et al. [10] was utilized for weather data based FST estimation. The energy balance approach was adapted for the model development. The total input energy in the fruit system that includes net short-wave radiation and net long wave radiation was balanced with total output energy, that includes emitted fruit thermal radiation, energy loss by evapotranspiration, sensible heat loss or gain, and total heat transfer within the fruit. The derived model utilizes four major weather parameters (air temperature, dew point temperature, solar radiation and wind speed) and the surrounding ground temperature to predict the mean apple FST. The open field weather data was acquired in real time from the agricultural weather network of Washington State University (WSU AgWeatherNet, available online: http://weather.wsu.edu/index.php). This network includes 186 automated weather stations located mostly in the irrigated regions of eastern Washington State [16]. Typically, each station collects 10 different weather parameters at a frequency of 0.2 Hz and provide mean value at every 15 min.
The process flow of weather based FST estimation is depicted in Figure 3. Firstly, the 'AppSense 1.0' application acquires longitude and latitude information of all weather stations in the network and utilizes the smartphone global positioning system (GPS) coordinates (longitude and latitude) to locate the nearest weather station. The FST monitoring tool downloads the weather parameters from the located station and applies the apple surface temperature dynamics model to estimate the FST.

Thermal-RGB Imagery Based FST Estimation
A customized image-processing algorithm was developed for estimating thermal-RGB imagery data based FST (Figure 4). The smartphone-integrated thermal-RGB imaging sensor (measurement range: −20-400°C, Accuracy: ±3°C or ±5%, focal length: 15 cm-infinity, thermal sensitivity: 70 mK) was capable of acquiring radiometric thermal (resolution: 160 × 120 pixels) and Red-Green-Blue (RGB) (resolution: 1440 × 1080 pixels) images ( Figure 4). Radiometric thermal data consists of pixelated temperature information.  Calibration of the thermal-RGB sensor was conducted using water [17] with an emissivity of 0.95-0.97, almost equal to that of plant leaves and apples surfaces. The water temperature was varied within a range of 10-50 • C in reference to typical FSTs. The actual temperature was measured using a liquid-in-glass thermometer for calibration of the thermal-RGB sensor-estimated temperatures. Five random circular spots were selected from the images and were analyzed for their mean temperatures in the sensor manufacturer provided Application (FLIR One, FLIR Systems Inc., Wilsonville, Oregon, OR, USA). The developed calibration equations were then used to correct the temperature data.
In this study, the pertinent RGB imagery data was used to identify and segment the apple from the rest of the canopy. The size of the RBG image was first resized to 160 × 120 pixels so that it could be overlapped with the thermal images for accurate pixel matching. Fruit color feature was used to separate the fruit from the stems, foliage, and background. For simplicity, the fruit shape feature extraction method was not employed as it is expected that the user will point the sensor to have several fruits prominent in the image.

Color Space Conversion and Fruit Segmentation
The YUV color space is one of the closest to the color perception of human eyes. The use of the YUV space may be more effective than RGB color space [18]. In YUV, Y represents the brightness of color space and can be very vulnerable to the influence of illumination. Hence, only U (red projection) and V (blue projection) color components were used for segmentation. Moreover, such color conversion makes the algorithm simple from a computational resource use point of view [19]. The RGB to YUV color transformation algorithm have been illustrated in Equation (1): The resulting YUV color space was then used to segment the fruit using the logic in Equation (2). Most of the segmented binary images (Figure 5b) had residual errors. Branches, stems and leaves in the frame were sometimes closer to the color features of maturing apples and can lead to such errors. A median Blur filter (size = 25 pixels), was applied to remove these errors (Figure 5c). This filter eliminated the small unwanted pixels from the frame. However, since only the largest apple in the image was the region of the interest for this study, the use of a fixed filter size was not completely effective and some parts of partially segmented apples remained in the region of the interest. To eliminate such areas, the maximum connected pixel domain was identified and the rest were rejected (Figure 5d). Figure 5e depicts the resulting segmented apple blob. Finally, the segmentation accuracy of 10 randomly selected sample images were estimated. The largest apple in the frame was manually labeled for each image and the segmentation accuracy of the algorithm was evaluated by dividing the area of the algorithm-segmented image with that of the manually segmented ground-truth image at each stage.

Image Overlapping and FST Estimation
A pixel offset was observed between the thermal and RGB image (Figure 5e,f). The eccentricity between two sensors often results in such an offset. The pixel offset was found to be about 40 pixels when the image was captured from a typical imaging distance of 20 cm. An image registration was performed between the thermal and RGB images to eliminate this offset. Finally, arithmetic operations were performed on the segmented thermal image (Figure 5h) to estimate the mean and maximum apple FST.

Field Tests
Field data was collected for four apple cultivars cv. Honeycrisp, Fuji, Gala and Red delicious. The orchards were at various locations within the state of Washington, USA (Table 1). First, 60 images were captured from an imaging distance of about 20 cm for cv. Honeycrisp using thermal-RGB imaging sensor integrated with an Android OS smartphone (model: OnePlus 6, OnePlus Technology Co. Ltd., Shenzhen, China) (Figure 6a). The image acquisition and on-board processing was performed using the 'AppSense 1.0' application to estimate FST. Moreover, actual surface temperature, or ground truth data was collected using a thermal probe (model: Thermapen ® Mk4, ThermoWorks, Lindon, Utah, UT, USA) of accuracy ±0.4 • C and response time of 2-3 s. Five sun-exposed apples were randomly selected in the orchard block and the thermal probe was punched into the upper 3 mm of the apple surface at five places to measure the surface temperature (Figure 6b). A stabilization time of 3 s was used for reliable temperature measurement. This data was used to test and refine the mobile FST monitoring tool and the functionality of the developed application. The resulting tool was further tested to monitor the FST of cv. Fuji (50 fruits), Gala (75 fruits) and Red delicious apples (20 fruits).  Figure 6. (a) Fruit surface temperature monitoring using developed tool and pertinent (b) ground-truthing using a thermal probe.

Data Analysis
The thermal-RGB imagery derived FST (FST i ), weather data derived FST (FST w ), ground truth measured FST (FST G ), and ambient air temperature (T air ) data was first checked for normality as a primary step in statistical analysis. A 'Pearson product moment' correlation test was then performed to test the relationship between FST i , FST G, and T air . Moreover, a 'two-sample t-test' was performed to quantify the mean difference between FST i and FST w . A similar analysis was performed to estimate the mean difference between FST i and FST G , FST i and T air , as well as imagery based maximum FST (FST i-max ) and FST G . A statistical analysis was performed in R-Studio ® (version 1.0.153-© 2009-2017, RStudio ® , Boston, MA, USA) and significance was inferred at the 5% level. Figure 7 depicts the representative segmentation results obtained for all four fresh market apple cultivars tested in this study. Within the 'AppSense 1.0' ecosystem, the developed algorithm provided a mean segmentation accuracy of 94.4%, 95.7%, and 97.5% at respective thresholding, blur filtering, and final apple blob extraction stages ( Table 2). The algorithm was able to segment matured apples from the background canopy attributes. Results indicated that different apple cultivars have some effect on the segmentation algorithm accuracy. The algorithm performed best for the cv. Red delicious as the dark and intense crimson color of fruit makes it less reflective and easier to distinguish from surrounding objects. Furthermore, the number of apples in the frame had no visual effect on the segmentation as only the maximum connected pixel domain was selected and the rest of the apples in the frame were filtered out. However, the apple glossiness that can produce a bright white spot on apples during imaging had an adverse effect on the segmentation due to color illusion (Figure 7e). Appropriate angle of the imaging sensor can help in avoiding these bright spots for accurate segmentation. Moreover, unsupervised classification-based techniques [12] and supervised machine learning techniques could be further explored for robust segmentation and are the subject of our on-going research.  The calibration results showed a strong correlation (R 2 = 0.99) and low measurement error (RMSE = 0.86°C) (Figure 8) between actual temperature and the thermal-RGB sensor measured temperature data. The statistical analysis of FST i and FST G for cv. Honeycrisp indicated that there was no significant difference in FST i-max (mean ± standard deviation) (38.8 ± 3.7°C) and FST G (39.3 ± 3.8°C

Fruit Segmentation
) (Two sample t-test, t 110 = 0.51, p = 0.6) (Figure 9). Ground truth measurements were taken at the upper neck area of the apples, as it was observed to be the hottest area on the fruit and therefore the most susceptible to sunburn. The close association of FST i-max and FST G suggest that the developed tool is reliable for FST monitoring in orchard conditions. There was a significant difference in mean FST i (30.4 ± 2.9°C) and FST G (t 110 = 12.5, p < 0.0001). This is arguably because FST i was a mean FST value of overall segmented apple whereas ground truth measurement targeted the hottest point of the apple.

Field Evaluation of the Mobile FST Monitoring Tool
A significant temperature difference was reported between the weather parameter derived mean FST (FST w ) (35.3 ± 3.4°C) and the imagery data derived mean FST i (t 110 = −9.61, p < 0.0001) for cv. Honeycrisp (Figure 9). The 'AppSense 1.0' application used data from an open field weather station that was located in the ranges of 4 to 7 km from the orchard sites (Table 1). The weather parameters from the closest identified weather stations might have large differences from the local microclimate in the test orchard sites. Such microclimate variation might have resulted in the FST w and FST i differences. A site-specific in-field weather station could assist in eliminating such errors and help estimate more accurate weather-parameter derived FSTs. Ongoing research in our lab is focused on in-field microclimate parameter-based FST estimation.
The FST i and air temperature (T air ) was significantly correlated with each other (Pearson product moment correlation, r = 0.66, t 54 = 6.4, p < 0.001) (Figure 10a). While comparing the FST and ambient air temperature (T air ), the mean FST i and FST w was significantly higher than T air (24.4 ± 2.3°C). Racsko and Schrader [3] and Shi et al. [12] have also reported a temperature difference of 5-18°C between air temperature and apple FST estimates. Analysis indicated a similar FST trend for cv. Fuji, Gala and Red delicious. A significant correlation was reported between FST w and FST i (r = 0.74, t 86 = 10.4, p < 0.001) (Figure 10b). Moreover, there was no significant difference between FST i-max (35.6 ± 2.4°C) and FST w (35.7 ± 2.7°C) (t 174 = −0.2, p = 0.84) ( Figure 11). However, there was a significant difference in FST w and mean FST i (31.8 ± 2.4°C) (t 174 = −10.7, p < 0.0001). Further investigation indicates that the mean FST i and FST w for cv. Fuji, Gala and Red delicious was significantly higher than T air (24.3 ± 1.8°C).

Conclusions
Followings are the conclusions from this study:

1.
The mobile FST monitoring tool performed reliably in field conditions to estimate FSTs of the four most common apple cultivars grown in the state of Washington. The developed smartphone application 'AppSense 1.0', successfully acquired thermal-RGB images and weather data and was able to perform real-time data analysis on-board the smartphone to estimate apple FSTs.

2.
The mobile FST monitoring tool measured maximum FST was consistent with ground truth measurements. A significant difference was recorded between weather and thermal-RGB imagery predicted FSTs, possibly due to the remote locations of the open field weather stations that were 4-7 km away from the experimental sites.
Overall, with further refinements and application deployment, the developed mobile sunburn prediction tool is expected to aid growers in accurate and real-time fruit surface temperature monitoring and actuation of subsequent control measures to avoid sunburn related crop losses. Such tools can also assist in monitoring other FST-influenced physiological parameters such as sugar and acid content, ripening process, fruit color and quality. Moreover, this tool has the potential to be customized for monitoring heat stressors in some of the sun-exposed perennial and annual specialty crops at produce maturation in the field.
Author Contributions: L.R.K. provided the idea of this work. L.R.K., B.W. and R.R. contributed to the conceptualization, software (Application) development and refinement of the user interface. L.R.K., R.T.P. and R.R. contributed in the experiment design. R.R. and B.W. contributed to the field data collection, data analysis and original draft preparation. L.R.K. and R.T.P. reviewed and revised the manuscript in the current form. Correspondence and request for the paper should be addressed to L.R.K. All authors have read and agreed to the published version of the manuscript.

Funding:
The research was performed at Washington State University through funding, in part, from NSF/USDA-NIFA Cyber Physical Systems program grant number 1837001 and USDA-NIFA WNP0745.