Design and Implementation of a Low-Cost, Linear Robotic Camera System, Targeting Greenhouse Plant Growth Monitoring

Abstract

Automated greenhouse production systems frequently employ non-destructive techniques, such as computer vision-based methods, to accurately measure plant physiological properties and monitor crop growth. By utilizing an automated image acquisition and analysis system, it becomes possible to swiftly assess the growth and health of plants throughout their entire lifecycle. This valuable information can be utilized by growers, farmers, and crop researchers who are interested in self-cultivation procedures. At the same time, such a system can alleviate the burden of daily plant photography for human photographers and crop researchers, while facilitating automated plant image acquisition for crop status monitoring. Given these considerations, the aim of this study was to develop an experimental, low-cost, 1-DOF linear robotic camera system specifically designed for automated plant photography. As an initial evaluation of the proposed system, which targets future research endeavors of simplifying the process of plant growth monitoring in a small greenhouse, the experimental setup and precise plant identification and localization are demonstrated in this work through an application on lettuce plants, imaged mostly under laboratory conditions.

1. Introduction

Nowadays, the use of dedicated, sensor-based growth status monitoring systems in cultivating greenhouse crops, while enhancing precision in controlling the crop environment, is regarded as a crucial and distinctive method in precision agriculture. This approach enables the efficient and accurate management of crop production, while also meeting the requirements for producing high-quality and safe agricultural products. Moreover, a meticulously controlled greenhouse environment effectively minimizes the unnecessary consumption of resources such as CO₂, water, and fertilizer, thereby enhancing the industry’s environmental sustainability [1]. The incorporation of an intelligent monitoring system enhances energy-saving and emission reduction outcomes while possessing the ability to forecast extreme greenhouse conditions [2]. Among other intelligent greenhouse monitoring techniques, various sensor imagers and integrated imaging modalities are employed in high-throughput phenotyping platforms to gather data for the quantitative analysis of intricate characteristics associated with plant growth, productivity, and resilience to biotic or abiotic stressors such as diseases, insects, drought, and salinity. These imaging methodologies encompass machine vision, multispectral and hyperspectral remote sensing, thermal infrared imaging, fluorescence imaging, 3D imaging, and tomographic imaging (e.g., X-ray computed tomography) [3]. Moreover, the use of artificial intelligence, big data, deep learning, the Internet of Things (IoT), and other cutting-edge technologies in greenhouse monitoring procedures are regarded as beneficial advancements that nowadays enhance unmanned greenhouse management, significantly facilitate plant growth monitoring, and control procedures [2,4].

Data analysis and evaluation tasks are becoming more and more common in precision agricultural research, while data acquisition tasks are supported by expanding computer vision techniques and access to data from various sensors [5]. A combination of data acquisition, data modeling, analysis, and evaluation is also carried out in the constantly evolving field of image classification and object detection based on Convolutional Neural Networks (CNNs) and Deep Convolutional Neural Networks (DCNNs). For example, a DCNN-based intelligence method for estimating above-ground biomass in the early growth stage for winter wheat is proposed in [6]. Another method that utilized digital images and the application of a CNN for growth-related monitoring traits, like the fresh and dry weights of greenhouse lettuce, was proposed in [7]. Moreover, some indicative yet successful application examples of the many developed and further optimized image processing algorithms and computer vision techniques that utilized CNN and DNN methods include the development of plant disease recognition models, based on leaf image classification [8,9], the simultaneous extraction of features and classification for plant recognition [10], simulated learning for yield estimation and fruit counting [11], automated crop detection in seedling trays [12], crop yield prediction before planting [13], weed detection [14], pest detection [15], automatic sorting of defective lettuce seedlings grown in indoor environments [16,17], tip-burn stress detection and localization in plant factories [18], and many others [19,20,21,22,23]. Deep learning techniques have shown significant advancements in machine and computer vision tasks concerning feature learning, thus facilitating plant data analysis and plant image processing while demonstrating great potential in the agricultural industry. The automatic features’ extraction from raw plant image datasets, for purposes such as prediction, recognition, and classification, involves the progressive formation of higher level image features to be composed from low-level features through the hierarchy [24]. These paradigms and the previously reported application examples outline a promising and non-destructive approach for treating massive amounts of data in various applications related to plants, including detection and localization, growth analysis, crop prediction, and plant phenotyping. The progress made in these areas highlights the potential of deep learning in advancing our understanding and management of plant-related processes.

However, deep learning-based algorithms, although often showing superior performance traits related to accuracy (e.g., compared with traditional approaches to computer vision-based tasks, like plant identification), require careful fine-tuning to achieve the highest desired outcomes. At the same time, a very large amount of generic data and available datasets is needed for pre-training and achieving the desired performance of the models [24]. Moreover, their success relies significantly on the availability of an appreciable amount of computing resources that can be utilized to successfully accomplish the assigned tasks. Furthermore, it is quite common for some approaches to utilize plant images during model training that exhibit either optimal imaging acquisition properties or disregard real field conditions. For example, these images might showcase a white or monochrome backdrop encircling the plant, the absence of soil surrounding the plant, or a meticulously prepared monochrome substrate devoid of any fertilizers near the plant. Additionally, these images may lack the presence of moss or depict plants that are not potted, among various other factors. Nevertheless, traditional machine vision, computer vision, and machine learning methods that rely on basic image processing techniques and fundamental machine learning algorithms (e.g., K-means, SVM, KNN, linear regression, random forest, etc.), especially when targeting smaller image datasets with fewer sensors utilized, still continue to be valuable, preferable, and valid for all the aforementioned factors [25,26,27,28,29,30].

Growers and researchers can reap several benefits by integrating machine vision, computer vision, and robotics for plant growth monitoring in greenhouses but also in the field [22,30,31,32]. For example, a constructed machine vision-guided lettuce sensing and features’ monitoring system (color, morphological, textual, and spectral) is illustrated in [33]. In [34], an urban robotic farming platform is demonstrated on a laboratory scale for lettuce recognition and subsequent water irrigation tasks. An integrated IoT-based, three-device-system of autonomous moving platforms, aiming to facilitate precision farming and smart irrigation by utilizing the data collected from many sensors, was introduced in [35]. A robotic platform with a vision system and a custom end effector, targeting the harvesting of iceberg lettuce under field conditions, is presented in [36]. In [4], a methodology is presented that combines the application of a digital camera and artificial intelligence for growth prediction as well as the harvest day and growth quality of lettuces in a hydroponic system. An automated growth measurement system for Boston lettuce in a plant factory is developed in [25]. Each lettuce was iteratively photographed, with the cameras moving in predefined increments along the planting beds. Plant growth curves were presented by the authors after applying image processing techniques for each plant’s contour detection and the subsequent estimation of each plant’s area and height. Similarly, agricultural tasks, like a flower harvesting mechanization, implemented as a three-axis design with mechanical parts based on a computer numerical control machine (CNC) and utilizing YOLOv5 deep learning techniques for the localization and detection of flowers, is presented in [37]. In [14], a small-scale Cartesian robot, aiming at performing functions like seed sowing, spraying, and watering in predefined locations, is presented. The system utilizes YOLO for weed detection. In a similar fashion, in [38], a small farming CNC machine for irrigation, planting, and fertilizer application in predefined locations is implemented. Moreover, the utilization of hyperspectral imaging sensors, along with the potential adaptation of other low-cost, yet promising multispectral compact designs [39,40] suitable for integration into robotic platforms, can be effectively utilized in agricultural applications to perform repetitive activities such as the plant photography that the present work focus on, but also facilitate other interesting agricultural procedures and applications such as fruit harvesting or even fruit ripening assessments.

However, a common challenge within the field of computer vision, particularly in the wider scope of agriculture, pertains to the accessibility of datasets for researchers to utilize in order to conduct tests and experiments while developing algorithms or devices. Creating custom image datasets is frequently required, a process that can be quite time consuming, lasting for hours or even days. Additionally, monitoring greenhouse plants and collecting data, especially for research purposes using basic hand-held cameras, can be a laborious, time-consuming, and monotonous undertaking. Moreover, to achieve the necessary accuracy and repeatability during plant image acquisition, the possibility of possessing a robotic mechanism that acts as a reliable and robust recoupment of a human photographer/crop researcher seems to be valuable. With the help of robotic cameras, plant images can be acquired at any time of the day, at the “exact same spot”, under almost identical conditions (e.g., same viewpoint, distance, and same shooting angle) without the need for a human’s presence in the field. The images can then be sent to a remote computer, and the researcher can further systematically monitor plants’ health and growth-related traits, with minimal effort.

With these factors in mind, the objective of this study was to develop an experimental, low-cost, 1-DOF linear robotic camera system tailored for automated plant photography. As an initial evaluation of the proposed system that targets, as an ongoing project, simplifying the process of plant growth monitoring in a small greenhouse in our university’s facilities, this study presents, in detail, the hardware materials used to construct the robotic experimental setup, the software developed for its operation, as well as the methodology followed to assess its construction and movement accuracy along with its ability to accurately identify and localize plants, demonstrated through the examination of lettuce plants primarily in laboratory settings. In this context, the present work focuses on top-view images of the plants, and most image processing and real-time lettuce detection operations are related to the task of color segmentation and the successful analysis of their total projected leaf area and contour, since the top-view images contain more information to monitor plant health than the side-view images [29,41].

Apart from this introductory section, the rest of the manuscript is organized as follows: Section 2 presents the proposed 1-DOF linear robotic camera system’s hardware design and construction in detail along with its software pipeline. Additionally, the experimental setup devised to assess the system’s movement accuracy and repeatability is thoroughly explained. Moreover, Section 3 demonstrates the results and the respective discussion over the conducted experiments related to the main aspects of the robotic camera system, and more specifically, its movement’s accuracy and repeatability assessment and plant identification and localization ability, while also discussing its limitations; it also provides related work from other research studies and a future research outlook. Finally, Section 4 concludes the manuscript.

2. Materials and Methods

2.1. System’s Design and Construction

A low-cost, 1-DOF linear robotic camera system, named RoboPlantDect, targeting the systematic image acquisition of plants and facilitating future research endeavors like greenhouse growth plant monitoring procedures, was fully designed and constructed from scratch (Figure 1a–h). The initial idea originated from the need for an automated plant photography system for lettuce cultivation research purposes in our university’s facilities, without having to sacrifice many human resources for field observations. The original purpose was to monitor, from above, lettuce seedlings, at the stage of four true leaves, growing in plastic pots and arranged in a single row inside an unheated saddle roof double-span greenhouse covered with polyethylene film. However, except from the thorough inspection and consideration of the usual design factors like the cost and the availability of the construction materials, the ease of assembly, and the maintenance costs, a few more prerequisites were defined early in the design process phase. More specifically, among others, these included the following:

• The system should be able to automatically detect and accurately localize the spaced plants that are growing apart in plastic pots, positioned in a fully randomized design, in a 3 m long row;
• The system as a whole, should be lightweight and portable but also easily scalable if needed in order to facilitate its easy positioning for monitoring other rows inside the greenhouse too;
• To ensure that the linear robotic camera system can precisely reach each of the target plants, the actuator must exhibit sufficient precision in its movement. Upon reaching the target, the system should capture a high-resolution, color, top-view image with the plant ideally centered;
• With none or minor modifications and/or software adjustment settings, the linear robotic camera system could also be applicable for other plant species’ monitoring research and investigation purposes in hydroponic systems (e.g., spinach, sweet peppers, herbs, and strawberries, to name a few);
• The system should not, in any manner, obstruct the plants’ growth;
• Detected and localized plant images along with all the plant growth-related data measurements performed, together with the datalogging information, should be automatically stored locally, but also transmitted to a remote location and synched in a secure and transparent manner;
• No human intervention should be necessary at any stage of plant detection and monitoring.

Figure 1. View and details of the constructed RoboplantDect 1-DOF linear robotic camera system. (a) Global system of coordinates, applied to the robotic camera system. Transformations and measurements involving the various coordinate systems of interest are performed in reference to this coordinate system, which belongs to the “right side” of the device. The camera is moving in the X-axis only. The left side is considered to be the other beam’s end, where the left tripod is secured; (b) panoramic top view of the system, moving towards the far left; (c,d) side views of the water-resistant plywood box cover, with sliding plexiglass window in front; (e) top view of the carriage; (f) system setup in the greenhouse, with lettuce arranged in a single row; (g) detailed view of the carriage’s bottom side, showing the camera, LEDs, distance sensors, and limit switches mounted; (h) top view, showing the system in “home” position, on the far right side.

Figure 1. View and details of the constructed RoboplantDect 1-DOF linear robotic camera system. (a) Global system of coordinates, applied to the robotic camera system. Transformations and measurements involving the various coordinate systems of interest are performed in reference to this coordinate system, which belongs to the “right side” of the device. The camera is moving in the X-axis only. The left side is considered to be the other beam’s end, where the left tripod is secured; (b) panoramic top view of the system, moving towards the far left; (c,d) side views of the water-resistant plywood box cover, with sliding plexiglass window in front; (e) top view of the carriage; (f) system setup in the greenhouse, with lettuce arranged in a single row; (g) detailed view of the carriage’s bottom side, showing the camera, LEDs, distance sensors, and limit switches mounted; (h) top view, showing the system in “home” position, on the far right side.

The implementation of the computer vision-guided system is divided into two parts, and more specifically, into the following: (a) the system’s design and construction, which describe and present the hardware parts and the implementation details that comprise the whole system assembly and (b) the system’s software programming andworkflow, which describes the system’s software building blocks and the complete pipeline. The detailed, full list of materials and parts that were designed, constructed, assembled, or purchased, is presented in

Table A1. List of materials and components.

Item #	Component	Short Usage Description	Quantity
1	Camera tripod	Whole structure support	2
2	V-Slot 2060 1000 mm: natural anodized	Horizontal support for the carriage	2
3	V-Slot 2060 1500 mm: natural anodized	Horizontal support for the carriage	1
4	Cable drag chain 10 × 10 mm (m)	Cable management	3
5	Drag/cable chain mounting bracket	-	1
6	Aluminum 248 × 40 × 2 mm	Cable drag chain support	2
7	Double tee nut	V-slot assembly	14
8	Spring-loaded tee nuts	V-slot assembly	10
9	Stepper Motor Driver TMC2209 V2.0	-	1
10	Plywood 250 × 150 × 6 mm	Carriage support	1
11	V-Slot Gantry Set 2020 × 2080 Xtreme	For the movement of the carriage	1
12	Stepper motor 2.8 kg.cm 42BYGHW208	-	1
13	Timing belt: GT2 (m)	-	4
14	Aluminum GT2 timing pulley: 12T—5 mm	Toothed pulley for the precise movement of the carriage	1
15	Power Supply 12 V 2 A: output 5.5 × 2.1 PSU-1602	-	1
16	Power supply 5 V 3 A for Raspberry Pi 4	-	1
17	Raspberry Pi 4-Model B-8 GB	-	1
18	Raspberry Pi 4 B Heatsink: black (set of 3)	RPi 4B cooling	1
19	DC Fan 50 × 50 × 10 mm 12 V: MagLev	RPi 4B cooling	1
20	Memory card microSDHC 64 GB Class 10: SanDisk Ultra	RPi 4B OS and programming	1
21	Microswitch Mini SPDT ON-(ON): with roller lever	Determination of the two ends of the carriage	2
22	Adafruit Time-of-Flight (ToF) sensor VL53L0X	-	1
23	Humidity and temperature sensor (AM2301)	Temperature, humidity	1
24	GPIO Header for Raspberry Pi 2 × 20 stackable	-	1
25	infrared LED board	Illuminates the plants in the infrared spectrum	2
26	Arducam stereo USB camera, synchronized, 2MP 1080P	RGB and IR images capture	1
27	PCB board electronics and other components	JST XH Connectors (2 pin/3 pin/4 pin), P2N222ANPN Transistor, resistors, capacitors, Relay DPDT 5 V, 1N4007 Diode, cables, multi-socket, bolts, nuts, washers, screws, cable insulation, etc.	1

of Appendix A. In Figure 2, a simplified schematic diagram shows the complete system’s pipeline.

2.1.1. Hardware: Stable and Moving Parts

The RoboPlantDect system was designed to monitor lettuce plants growing in a greenhouse inside plastic pots and further arranged in 3 m long rows. A prerequisite design definition was that the whole construction should be lightweight and that it could be placed above any row of plants inside the greenhouse, by a person or two, without disassembling the whole device each time. Following this requirement, the main structure of the system was assembled mainly by combining high quality, lightweight linear aluminum extrusions.

For this purpose, three V-Slot 2060 ($20 \mathrm{m}\mathrm{m}\times 60 \mathrm{m}\mathrm{m}$) aluminum beams [42] (Figure 3c) were connected in series, one 1.5 m and two 1 m long, since the desired 3.5 m long single beam could not be found at the time of device’s construction at the local market. By placing the 60 mm side vertically, no bending was observed along the whole run of the 3.5 m long rail. V-Slot’s aluminum profile construction characteristics feature the internal V-shaped channels and V-grooves that further permit the linear V-Slot wheels’ smooth movement (Figure 3c,d). For firmly connecting the V-slot 2060 in series, tee nuts were placed in the top and bottom ends of the V-Slot connections at both sides (Figure 3d).

Moreover, at both edges of the 3.5 m long beam, two camera tripods (Figure 3a) [43] capable of extending up to 210 cm height were attached to support the whole structure. These tripods are very lightweight, and they significantly facilitate height adjustments. They also have the added benefit of being able to be positioned in any uneven area, which is a typical sight in greenhouses. Finally, at the top of each tripod, a mounting plate ¼ was placed (Figure 3b) to ensure that both tripods were fitted to the aluminum beams securely.

A cable drag chain $10 \mathrm{m}\mathrm{m}\times 10 \mathrm{m}\mathrm{m}$ organizer (Figure 4a) was placed on top of the beams to ensure the proper grip and smooth movement of the power cable. Made from nylon, with a bend radius of 18 mm and external dimensions of $17 \mathrm{m}\mathrm{m}\times 14 \mathrm{m}\mathrm{m}$, it proved to be an ideal choice for cable management. To ensure that the cable organizer always stayed on top of the V-slot beams and did not fall on either side (during system movement), two pieces of a $1 \mathrm{m}\mathrm{m}\times 40 \mathrm{m}\mathrm{m}$ aluminum beam were cut, drilled, and further used as both protective sidewalls and as a guide (Figure 4b). The latter aluminum supports were secured with spring-loaded ball nuts (Figure 4c). This prevents unwanted movement of the nut in the slot and allows the nuts to be inserted in a vertical position, as they will not slip due to gravity. As a carriage for the main construction platform, a V-slot Gantry Plate set of $20\text{–}80 \mathrm{m}\mathrm{m}$ (Figure 4d) [44] was used and under it, a piece of marine plywood (dimensions $\left(W\times L\times H\right):250 \mathrm{m}\mathrm{m}\times 150 \mathrm{m}\mathrm{m}\times 6 \mathrm{m}\mathrm{m}$) was secured. Finally, another small piece of plywood beam (dimensions $\left(W\times L\times H\right):35 \mathrm{m}\mathrm{m}\times 115 \mathrm{m}\mathrm{m}\times 20 \mathrm{m}\mathrm{m}$) was fixed to the carriage to link the carriage with the drag chain and the cables. The system’s power switch was attached to this beam. To protect the carriage (and all the electronic components mounted on it) from the accidental use of the greenhouse’s mist irrigation system, a water-resistant plywood box was designed, constructed, and used as the system’s protection cover (Figure 1c,d). The final construction turned out to be robust, adaptable, and dependable.

The reference system of coordinates $\left\{X_w,Y_w,Z_w\right\}$ is considered attached firmly to the right tripod, and its origin ($X_w=Y_w=Z_w=0$) is shown in Figure 1a. Every linear movement of the 1-DOF camera system (i.e., translation along the X-axis) is performed in relation to this reference system. The linear movement is converted in the software into the corresponding motor’s steps, so as to precisely instruct the stepper motor to move the carriage platform, and subsequently, the whole cameras’ system structure, to achieve the desired pose.

2.1.2. Hardware: Motor, Controller, and Transmission

The movement of the carriage was performed using one high-torque, bipolar stepper motor (12 V DC, 0.4 A/phase, 200 steps/revolution, size NEMA17) with a 1.8° full step angle [45]. The stepper motor was positioned beneath the carriage and oriented with the shaft pointing upwards in order to link it with the timing belt and create a linear rotor drive system (Figure 5a).

A GT2 aluminum timing belt pulley with 12 teeth was attached to the shaft of the stepper motor (Figure 5a). Its small diameter retained the torque produced by the stepper motor. In addition, a 3.5 m long GT2-type timing belt (2 mm pitch, 6 mm width, made of neoprene and fiberglass materials) was attached to the two ends of the beam, which was used to transmit motion. As the axis of the stepper motor rotated, the timing pulley forced the carriage to move across the beam. The control of the stepper motor was achieved with a TMC2209 V2.0 [46] stepper motor driver connected to the Raspberry Pi and used a microstepping factor of 2 (e.g., half step), resulting in 400 steps per revolution of the motor shaft (Figure 5b). The calculated steps per mm for the linear motion produced by the belt was found to be $\frac{steps}{mm}=\frac{S_{rev}·f_m}{p·N_t}=\mathrm{16,667} \mathrm{s}\mathrm{t}\mathrm{e}\mathrm{p}\mathrm{s}/\mathrm{m}\mathrm{m}$. Here, $S_{rev}$ is the number of steps per revolution of the motor, $f_m$ is the microstepping factor, $p$ is the pitch, and $N_t$ is the number of teeth on the pulley attached to the motor shaft.

2.1.3. Hardware: Sensors, Auxiliary Electronics, and Raspberry Pi

Two SPDT momentary microswitches (model KW4-Z5F150, YueQing Daier Electron Co., Ltd., Yueqing City, Zhejiang, China [47]) with roller levers were used as limit switches (Figure 6a). They were attached at the far left and right of the carriage, so that the carriage could detect the origin and the end points of the line, respectively. When the carriage reached the end of the route, the corresponding microswitch contacted the tripod. In this way, the system determines the start (“home” position) and end positions of the route on the beam. An Arducam synchronized stereo USB camera [48] was used to capture the images (Figure 6e) in the current design. This module consists of two fixed focus cameras integrating OV2710 OmniVision (Santa Clara, CA 95054, USA) CMOS sensors [49] with the same resolution of 2 MP and sensitivity to both visible light (RGB) and infrared light (IR). In this work, the RGB camera was mainly used for the plant recognition task. The eight LEDs installed on the module board are infrared LEDs and are automatically turned on when the IR camera is used.

This stereo USB camera was chosen based on its synchronization capabilities, as well as its similar characteristics and features to both RGB and IR cameras. If images corresponding to the same scene are captured by the two cameras, they can be further combined with a previous image registration processing method to obtain the relevant Normalized Difference Vegetation Index (NDVI) images of the plants, thus obtaining useful data on the health the plant itself. However, the NDVI-related calculations are reserved for future work. To ensure the sufficient illumination of the plants during image capture also in the infrared spectrum, two additional infrared LED boards (3 W high power 850 infrared LED, 100° FOV, [50]) were used (Figure 6d). The built-in, adjustable resistor was adjusted to allow the LED to turn on in all ambient lighting conditions. A 1 m long LED strip (60 white LED 3528/m, input voltage 5 V DC) with the USB plug removed was also used complementarily to ensure the sufficient illumination of the plants in the visible spectrum.

In this study, two different low-cost distance sensors were used to provide complementary distance measurements during image capture and as the carriage moved linearly over the plants. The first was a VL53L0X sensor module [51], which contains a 940 nm laser source and a matching sensor that can detect the “time of flight” (ToF) of the light bouncing back to the sensor (Figure 6b). This distance sensor supports two modes of operation: normal mode, where the maximum measurable range is about 1200 mm, and long-range mode, where the maximum measurable range is 2100 mm. The results are noisier in the latter case [51]. Both modes provide the same output data precision, that is, 1 mm. In the present study, this distance sensor was programmed to operate in normal mode and demonstrated its accuracy, particularly for distance measurements within the range of approximately 600 mm (Figure 7a,c), The detection angle was 25 degrees.

An additional inexpensive sensor with a slightly wider detection angle of 30 degrees was also employed to obtain distance measurements by combining the readings from both sensors. The HC-SR04 ranging module (Figure 6c) [52] includes an ultrasonic transmitter, a receiver, and a control unit for determining the distance to an object. This sensor measures a range from 2 cm to 400 cm with an accuracy of 0.3 cm. The camera and the sensors were positioned in very close proximity to one another. For the purposes of this work, a specific experiment was performed to test the calibration process of both distance sensors, while examining the possibility of the existence of linear dependence in the datasets with those of manual measurements (true height). In this study, a sample (3D-printed box with dimensions $\left(W\times L\times H\right):89.7 \mathrm{m}\mathrm{m}\times 99.6 \mathrm{m}\mathrm{m}\times 29.7 \mathrm{m}\mathrm{m}$) was placed sequentially every 1 cm at a distance ranging from approximately 225 mm to 700 mm below the lens of the RGB camera and the sensors (Figure A2a, of Appendix A), and 48 RGB images were captured. The sensors were employed to calculate the distance of the sample from each sensor. The purpose of this experiment in the present work was twofold: The first was to test the calibration process of both distance sensors and to establish a relationship of their outputs as compared to the ground truth height by performing measurements in a range that matches the final application (Figure 1f). The second was to gather insight regarding the scale conversion of pixels to physical measurement units such as millimeters (mm), which are necessary for analyzing measurements like the lettuce diameter or the total projected leaf area. By analyzing the object in all these images (e.g., by simple averaging the distance in pixels of both different sides of the sample object, as appear in the images versus the distance in reality in millimeters), the scale ($\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}/\mathrm{m}\mathrm{m}$) as well as the factor (${\mathrm{m}\mathrm{m}}^2/\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$) along the various distances/depths could be estimated (Figure A2b, of Appendix A). From these calculations, the resolution can be approximated. Furthermore, accurate distance calculations for a lettuce plant as depicted in an image (such as its maximum diameter) can be achieved with reasonable precision, assuming that the two extremity control points on the leaves that are utilized for the maximum diameter calculation and belong to its contour’s perimeter lie nearly in the same focal plane, such that they appear approximately at a similar distance from the camera. However, all measurements related to the lettuce diameter and its area in this study are presented in pixels. Although this approach does not provide absolute measurements in metric units, it continues to be valuable, since it enables the relative comparative analysis of lettuce plants, since they are monitored using the same imaging system.

Nevertheless, prior to fitting a specific model to the acquired data using the 2 sensors, we initially investigated if a linear relationship could be established. For this purpose, correlation coefficient matrices $\left(r\right)$ were formed, considering a pairwise variable combination of the average readings of 30 measurements per height distance acquired with each sensor (all results converted to mm) and the manual measurements as well. Since the three correlation coefficient matrices that were formed (

Table 1. Correlation coefficient matrices between sensors and manual measurements.

Ultrasonic-True (r)	ToF-True (r)	Ultrasonic-ToF (r)
$r_1=\left[\begin{array}{cc}1& 0.9998\\ 0.9998& 1\end{array}\right]$	$r_2=\left[\begin{array}{cc}1& 0.9983\\ 0.9983& 1\end{array}\right]$	$r_3=\left[\begin{array}{cc}1& 0.9984\\ 0.9984& 1\end{array}\right]$

) were too close to 1, there was a strong indication that the measurements acquired with the different sensors in pairs could be linear correlated.

In Figure 7a,c, the scatter plot shows the relation between the respective measurements of height using each distance sensor, their simple average, and the best fitted line after applying multiple linear regression with an interaction term (also depicted in Figure 7b), as compared to the true height measured manually. The data measured with both sensors were treated as predictors and the true height as the response. The regression coefficients for the linear model that was utilized in this study are shown in Equation (1) below.

$$y=12.9749+0.2509·x_1+0.8177·x_2-{10}^{-4}·x_1·x_2,$$

(1)

The maximum of the absolute value of the data for this model (maximum error) was calculated to be $max.error=3.41\times {10}^{-13} \mathrm{m}\mathrm{m}$, which is a sufficiently small value and much smaller than any of the sensors’ data values used, indicating that this model accurately follows the measured data of both sensors. Moreover, the fit to the data had the same value for both the correlation coefficient $R^2$ and the adjusted correlation coefficient $R_{adjusted}^2$, with $R^2=R_{adjusted}^2=0.9999$ and root mean square error $RMSE=1.2226 \mathrm{m}\mathrm{m}.$ Thus, we conclude that according to the experimental height measurements and the analysis performed, the linear model as described in Equation (1) above fulfills the distance measurement requirements for the purposes of the current work.

To control the fan (mentioned below), the LEDs, the infrared LEDs, and the relay that controls the stepper motor driver, 2N2222A NPN transistors were used, which can handle the current required for these devices and the necessary resistors. An ordinary 12 V DC 2 A power supply was used to power the stepper motor and cooling fan. Finally, to connect all the necessary devices and electronic components, a simplified electronic circuit (Figure 8a) and a printed circuit board (PCB) (Figure 8b) were designed using EasyEDA software ver. 6.5.20 (Figure 8a). JST male/female connectors (JST XH 2.5 mm) were used to connect all devices to the circuit board. The control of the relay and the stepper motor driver, the control of the distance sensors, the limit switches, and the infrared LEDs, as well as the image acquisition, processing, and storage were carried out by a computing unit small enough to fit on the cart and powerful enough to handle all the tasks required. The Raspberry Pi 4 - Model B - 8 GB [53] proved to be ideal because, among other things, it supports WiFi, integrates USB ports, and has sufficient computing power and 8 GB of RAM to process the captured images.

The official Raspberry Pi USB-C power supply (output voltage: 5.1 V DC, max. power: 15.3 W) was used to power the Raspberry Pi and all peripheral sensor devices. Given the high temperatures that can be achieved in a greenhouse, heat sinks coupled with a 12 V fan (the latter to also move the dust away) were used to cool the Raspberry Pi (RPI) (Figure 8c).

2.2. System’s Software Programming and Workflow

2.2.1. NIR and RGB Cameras’ Calibration

The geometric calibration of both cameras was conducted by assuming a pinhole camera model and a frontal parallel arrangement. This calibration process involved the utilization of a known-size asymmetric checkboard pattern consisting of 22 × 15 squares, with each side measuring 25 mm. The checkboard pattern was supported by a wooden flat surface, which was securely mounted on a robotic platform (Figure A1b, Appendix A). The entire setup was positioned at a distance from the cameras’ lenses that approximately matched the expected maximum height of the potted lettuces in the final application, ranging approximately 35 to 70 cm. Next, a total of 15 pairs of images were taken, with each pair consisting of a left and right image. These images were captured from different 3D positions and orientations, ensuring that the inclination of the checkerboard did not exceed 35 degrees relative to the plane of the cameras during each image pair capture. The images were saved in an uncompressed format, such as a PNG. The resulting dataset consisted of 30 images, which were then processed using the stereo camera calibration tool in Matlab^TM, Computer Vision Toolbox^TM. This processing allowed for the calculation of various parameters related to each camera’s intrinsic parameters, distortion coefficients, and extrinsic camera parameters. These parameters included the radial and tangential distortion, intrinsic matrix, focal length, principal point, and mean reprojection error. The NIR camera’s translation and rotation matrices, as well as the fundamental and essential matrices, were computed in relation to the RGB camera. These computed values were then compared to the manufacturer’s datasheet (Arducam, OV2710 Dual-lens module) [54] whenever possible.

Table A2. Stereo camera calibration parameters.

Camera	Calibration Parameters	Equation	Calculated Values
RGB NIR	Intrinsic parameters *	$\mathrm{K}=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$	$K_{rgb}=\left[\begin{array}{ccc}1051& 0& 1026.2\\ 0& 1052.7& 498.223\\ 0& 0& 1\end{array}\right]$
			$K_{nir}=\left[\begin{array}{ccc}1060.4& 0& 945.3329\\ 0& 1062.5& 513.1052\\ 0& 0& 1\end{array}\right]$
RGB NIR	Radial distortions	$x_{dist}=x·(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)$	$k_1=-0.0460$ $k_2=0.1395$ $k_3=-0.1193$
		$y_{dist}=y·(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)$	$k_1=-0.0391$ $k_2=0.1365$ $k_3=-0.1273$
RGB NIR	Tangential distortions	$x_{dist}=x+\left[{2p}_{1}xy{+p_2(r}^2+2x^2\right)$] $y_{dist}=y+[{2p}_{2}xy{+p_1(r}^2+2y^2)$]	$p_1=0.000055984$
			$p_2=-0.0003043$
			$p_1=-0.00016$
			$p_2=-0.000077077$
RGB NIR	Rotation of RGB camera relative to NIR camera	$\mathrm{R}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]$	$\mathrm{R}=\left[\begin{array}{ccc}1& 0.000141114& 0.0021\\ -0.00013571& 1& -0.0026\\ -0.0021& 0.0026& 1\end{array}\right]$
RGB NIR	Translation of RGB camera relative to NIR camera **	$T=\left[\begin{array}{ccc}{t}_1& t_2& t_3\end{array}\right]$	$T=\left[-\begin{array}{ccc}14.4076& 0.2578& -1.3885\end{array}\right]$

* units in pixels, ** units in mm, $p_1,p_2: \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n} \mathrm{c}\mathrm{o}\mathrm{e}\mathrm{f}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}$, $k_1, k_2,k_3: \mathrm{r}\mathrm{a}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{l} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n} \mathrm{c}\mathrm{o}\mathrm{e}\mathrm{f}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}$, and x,y: location of undistorted pixels (measureless).

of Appendix A presents some of the noteworthy parameters that were calculated. Accurately determining the pose (position and orientation) of the NIR camera with respect to the RGB camera is crucial for performing NDVI calculations, especially considering that both fixed cameras have almost identical technical specifications. The proper alignment of the RGB and NIR images is essential to ensure the integrity of the data obtained, even though NDVI calculations are not the focus of this study. Ultimately, the resulting overall mean reprojection error of 0.17 pixels was deemed acceptable for the current scenario.

2.2.2. Software: System Pipeline

The user interface of RoboPlantDect is designed to be intuitive and lightweight. It utilizes the cvui2.7.0 UI library [55], which is built on OpenCV drawing primitives. Through this interface, users can easily perform various tasks, as shown in Figure 9a. One of its key features is the ability to set the scheduled time interval for plant recognition and image acquisition, along with specifying the number of plants to be identified. The system is designed to perform several tasks repeatedly.

Firstly, it identifies the plants and records their precise locations. It then captures images of these plants using both RGB and NIR cameras while also recording the distance of each plant from the cameras based on readings from both distance sensors. Next, the system processes the captured images to determine the total projected leaf contours, diameter, center, total projected leaf area, and centroid of each plant. This information is then combined with the date and time of the recording. All these calculated data values, along with the date and time, are logged into a logfile for future reference. Additionally, the system offers the option to overlay this information onto each image. Furthermore, the acquired RGB and NIR images are saved separately in designated folders.

To ensure data accessibility and backup, the system synchronizes the folders containing the images and recorded data with other dedicated computers through cloud storage. This allows for easy sharing and collaboration among multiple devices.

2.2.3. Plant Finding

A series of procedures must be conducted to identify the plants (see also the flowchart in Figure 10):

• The initial step in the process is crucial as it involves determining the starting position of the carriage. To achieve this, the carriage is moved automatically to the far-right edge of the beam until the right end switch is activated. This position is then labeled as “position zero” (home position).
• Subsequently, the user must have indicated the number of plants for identification. A plant counter is then established and employed to ascertain the number of procedural iterations that will be carried out. In instances where the number of plants remains constant (as is typically the case in practical scenarios, where a specific number of plants exist in a row), this step is omitted, and the number of plants remains unchanged.
• The RGB camera that is utilized for plant recognition is then initialized. An image of $W\times H: 1920\times 1080 \mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$ is captured for further processing.
• Since the RGB camera has a wide field of view (95° diagonal), it is necessary to crop the image to ensure that only one plant (and potentially small segments of other plants) is present. This operation results in a square frame (cropped image) that features the same center as the original image. This not only helps in better focusing on a single plant, but also reduces the image processing time for the smaller image; the amount of cropping in pixels (a border with the same number for left and right sides as well as the up and down sides) can be determined by the user through dedicated buttons in the GUI (left side in Figure 9a). If ${CF}_w$ denotes the amount of pixels for cropping the image on the left and right sides (width dimension), and similarly, if ${CF}_H$ denotes the amount of pixels for the top and bottom sides (height dimension), the following relationship holds: ${CF}_w-{CF}_H=420.$ For example, to result in $888\times 888 \mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}$ cropped images, ${CF}_w=516$ and ${CF}_H=96$ pixels, respectively. (In this study, the experiments were conducted using cropped images of dimensions $360\times 360 \mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$ and $888\times 888 \mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$).
• To filter out any high-frequency noise that may exist, a Gaussian blur (with 11 × 11 kernel size) is applied to the image.
• The image is converted to HSV color space.
• The automatic identification of the plant primarily relies on analyzing its color. To achieve this, users are required to predefine specific thresholds for the Hue (H), Saturation (S), and Value (V). These thresholds determine the range of colors that will be considered as the plant’s color. In this process, the undesired colors are masked out, resulting in a black color in the empty space of the image. To facilitate this procedure, a dedicated menu called “Color Picker” has been incorporated into the graphical user interface (GUI). By pressing the “Tweak Parameters” button, the user can access the “Color Picker” window, where they can adjust the sliders to carefully select the appropriate upper and lower thresholds for the Hue, Saturation, and Value. This ensures that only the colors of the plant are retained, while all other colors are filtered out, leaving a black background. Through an empirical analysis, specific lower and upper thresholds were determined for the examination of lettuces. The chosen thresholds, which proved to be effective and remained unchanged for all the experiments in this study, are as follows: $H\in \left[23, 84\right]$, Saturation $S\in \left[30, 255\right]$, and $V\in \left[70, 255\right]$. Another crucial setting is the exposure value, which can range from 0 to 255. The appropriate value should be selected based on the lighting conditions. In the current application, a value of 200 was chosen during the experiments, which remained unchanged too (the installed LEDs used for illumination proved to be sufficient during all experiments). Additionally, a real-time video is displayed below the sliders to provide users with a convenient visual representation (Figure 8b).
• Having available the upper and lower thresholds for the color values in the HSV space from the previous step, the algorithm proceeds by tracking a particular color (i.e., green) in the image by defining a mask using the inRange() method in OpenCV. After this, a series of erosion and dilation operations are performed to enhance the solidity and distinguishability of the plants, which may result in the appearance of small blobs in the images. For the erosion morphological operation that is performed twice, a $3\times 3$ (8-neighborhood) structuring element is applied. The dilation operation is performed once with a similar structuring element. The objects present in the image are determined using the findContours() function in OpenCV. In case no objects are found, the process is repeated from the beginning, with a slight shift of the carriage to the left (equal to the number of steps set by the user as Jump Steps).
• If multiple contours are detected, the contourArea() function in OpenCV is applied to discard the smaller ones, retaining only the largest contour. If the identified contour is deemed too small to represent a plant, the process is repeated, with a slight shift of the carriage to the left (empirically set to 2000 steps).
• Once a contour that potentially represents a plant is identified, the minEnclosingCircle() function is utilized to obtain the coordinates of the circle’s center and its radius. This information is then used to draw a circle around the plant.
• If the center of the plant does not align with the center of the image, the carriage is shifted either to the left or right, aligning the camera precisely with the plant (by means of bringing the camera exactly on top of the plant). Subsequently, the entire process is repeated from the beginning. Otherwise, the position of the carriage (and consequently, the plant’s position) is recorded in a file, and the plant counter is increased by one.
• If there are no more plants to be searched for, the procedure concludes.

Figure 10. Flowchart of plant finding procedure.

Figure 10. Flowchart of plant finding procedure.

2.2.4. Image Acquisition, Plant Identification, and Localization

After the plant recognition and the recording of their positions, several measures are implemented to acquire images of the plants and analyze the data collected from the diverse sensors (Figure 11). If the Autofind button has been initially activated, the Find_Plant() function is executed to locate the plants before capturing the final images. The Photo_Sequence procedure will attempt to locate the specified number of plants as indicated on the control panel (number of plants set by the user). In case fewer plants are detected under the moving system, the process of finding plants will be repeated. If, once again, an insufficient number of plants is detected, the finding plants sequence will be repeated for the final time. If the desired number of plants is still not found, the photo sequence will proceed with capturing images of the plants that were detected. Upon completion of the Photo_Sequence, the LED strip will begin to flash, alerting the user to the issue. To resolve the error, the user can press the corresponding button on the control panel (“Clear Error” button, Figure 9a), thereby deactivating the LED strip.

Otherwise, a new folder is created with the current date and time as its name. The positions of the plants are retrieved from the relevant file (this function is useful for predefined targets). The carriage is then moved to the next plant’s position. Subsequently, both the NIR and RGB images are taken from the exact same spot after positioning the carriage appropriately. In the RGB image, the plant is identified, and various measurements such as the total projected leaf contour (perimeter), the centroid and total projected leaf area (PLA), etc., are computed and displayed. The carriage is repositioned multiple times in different locations around the identified central point of the plant, with the aim of collecting multiple measurements using the distance sensors. Subsequently, the weighted distance, as described by the linear model in Section 2.1.3 (Equation (1)) between the plant and the sensors, is computed. This information can be utilized further for the plant dimensions (e.g., maximum diameter) calculation. For example, in order to provide a reasonable approximation of the lettuce width (maximum diameter), the simple geometric scenario as shown in Figure A3 of Appendix A can be used. Following this and the triangulation principle, an approximation of a plant’s diameter (D) in metric units (mm) is given as follows: $D=(P·L)/f$, where in the equation, the letter P denotes the plant’s width as it appears in the RGB imaging sensor in pixels, f stands for the focal length of the camera in mm (i.e., $3\pm 0.5\% \mathrm{m}\mathrm{m}$, [48]), and L represents the distance between the camera/sensors and the plant (in mm), as provided by Equation (1). The pixel size is known from [48] to be $3 \mathsf{\mu }\mathrm{m}.$ The procedure, “get data from sensors and compute the plant’s dimensions”, as depicted in the Photo_Sequence flowchart in Figure 11 below, relies on both distance sensors to acquire meaningful measurements at present. This information will be enriched in a future upgrade of the system by fusing data also from the stereo imaging pair. Nevertheless, for the purposes of the present work, dimensions related to growing plants’ parameters are provided in pixels.

Various images including the original, cropped, and images with calculated data overlaid, are saved in the previously created folder. This entire process is repeated until all the available plants have been identified and localized. Currently, the predetermined maximum time that can be devoted for each plant identification is established at a specific value, which has been determined through empirical methods to include a margin of error ($2\pm 0.5 \mathrm{m}\mathrm{i}\mathrm{n})$. Should the system exceed this threshold, it is deemed a failure and proceeds to identify the subsequent plant. In a usual scenario, the time to detect a plant is by far below the minute.

2.3. Experimental Setup Evaluation and Measurements on Plant Samples

2.3.1. Experimental Setup for System’s Accuracy and Repeatability Evaluation

To assess the precision of the RoboPlantDect’s positioning and its ability to navigate above plants, as well as to evaluate its overall construction, a specific experimental procedure was conducted in the laboratory prior to any other experiments. This experimental setup aimed to evaluate the accuracy and consistency of the robot’s camera positioning along the linear X-axis, as well as assess its construction and mechanical integrity by detecting any misalignments or deviations along the Y-axis, even though the system does not move along this axis. Additionally, adjustments were made to the tripods to ensure that the aluminum beams were parallel to the floor, thus achieving proper positioning along the Z-axis. The evaluation procedure involved repeating the system’s positioning on the same predefined 15 targets for a total of 10 times, allowing for an assessment of the accuracy and repeatability.

The KY-008 laser transmitter module, which emits light at a wavelength of 650 nm and operates at a voltage of 5 V, was installed in a vertical position on the carriage and connected to the Raspberry Pi. A measuring tape was securely fixed on the surface of a CNC cut plywood sheet measuring $\left(W\times L\times H\right) 3000 \mathrm{m}\mathrm{m}\times 50 \mathrm{m}\mathrm{m}\times 3 \mathrm{m}\mathrm{m}$. This plywood sheet was positioned above 5 empty pots and maintained parallel to the ground at all times, as shown in Figure 12a. The origin of the measuring tape was aligned with the axis origin of the “home” position. In addition to the KY-008 module, a commercial tripod-mounted green laser beam with a wavelength in the range of 510–530 nm was placed near the “home” position. The horizontal cross beam of this laser beam was precisely aligned in parallel with the measuring tape, as depicted in Figure 12a,b.

The experiment involved setting 15 targets in incremental steps and recording their positions relative to the “home” position. Throughout the experiment, the red laser beam continuously emitted towards the area of the measuring tape. This setup was repeated for 10 iterations. At the beginning of each iteration, the carriage was set to the “home” position and then moved to each target, simulating a positioning task for the purpose of identifying and photographing plants. The 2D projected red spot of the laser beam on the measuring tape was recorded when the carriage reached each target sequentially. An evaluation of the overall performance of the robotic system was conducted by measuring the distance between the coordinates of the actuator and each target point. Furthermore, the distance between the two vertical laser beams (red and green beams) was manually measured and recorded for each target.

2.3.2. Laboratory Experimental Setup on Lettuce Plants: Image Acquisition and Evaluation Metrics

As an initial evaluation of the system, lettuce seedlings (Lactuca sativa var. Tanius) with four true leaves were transplanted into plastic pots with a capacity of 10 L. These pots were spaced apart in a fully randomized manner in single rows that were approximately 3 m long, with 5 or 6 pots per row. Two experiments were conducted in the laboratory, with both taking place between April and May 2022. In the first experiment, during which the pots containing the plants remained stationary, six baby lettuce seedlings (named A to F) were subsequently detected, localized, and photographed once on the same day and for 15 iterations. The second experiment was conducted in a 4-week interval, in which 5 different lettuce plants (named A to E) were detected, localized, and photographed once every week (growth days 10 to 31 after transplanting). The spacing between the consecutive pots was again fully randomized throughout this specific experiment. In other words, the plants were randomly shuffled and positioned in a row before the experimental detection and photographical procedure each time in this 4-week interval. The plants were watered daily under all circumstances and manually received a nutrient solution in the laboratory. All measurements were carried out between 10:00 am and 14:00 pm. The LEDs of the device used for illumination proved to be sufficient during both experiments.

The degree of detection between the projected leaf area of the lettuce in the ground truth image, which is manually traced to outline the lettuce contour (manual contour delineation), and the corresponding leaf area determined and calculated using the Identify Plant procedure was evaluated using the Intersection Over Union (IoU) metric (Figure 13). This metric provides a ratio that represents the extent of overlap between the projected lettuce area detected in the image and the ground truth area relative to the combined area of both the detection and ground truth. The IoU values range from 0 to 1, where a value of 0 indicates no overlap and a value of 1 indicates a perfect overlap. To calculate the projected leaf area as depicted in the images, all the pixels within the plant area that fall within the identified or manually traced plant contour are considered. The contours are treated as convex polygons. The total projected leaf area for each plant is obtained by summing the product of all the pixels within the plant area and the pixel/mm ratio, if available. The metrics Precision, Recall, and F1 score were also calculated. The Precision metric is derived by dividing the intersection area of each pair by the ground truth area, whereas the Recall metric is computed by considering the intersection area of each pair with the detected area (Figure 13). The F1-score metric is calculated as the weighted average of the Precision and Recall as depicted in Equation (2) below. Additionally, for each defined contour, the centroid was computed. The manual tracing of the lettuces’ contours in the images of the experimental datasets was accomplished using ImageJ ver. 1.53t software (National Institutes of Health, Bethesda, MD, USA).

$$F1\text{-}score=\frac{2\times Precision\times Recall}{Precision+Recall},$$

(2)

In the second experiment, the morphological characteristics of the plants are further examined by considering certain parameters. These parameters are illustrated in the tables and the images provided. Specifically, the Euclidean distance (in pixels) between the centroids and the centers of the minimum enclosing circles is presented for both the experimental and manually traced contours. Moreover, the diameters (in pixels) of these circles are considered, along with the stockiness and compactness (detailed in Section 3.3). Finally, the overall performance and the relative errors are reported.

3. Results and Discussion

The subsequent subsections provide an overview of the performance assessment of the suggested system, encompassing the primary experiments conducted in the laboratory and specifically focusing on the essential metrics. These metrics include the following: (a) the accuracy of the robotic system’s motion, (b) the successful identification and localization of plants, and (c) the determination of various morphological attributes of the plants that are used commonly in plant phenotyping applications for plant growth qualification, such as their diameter, as well as contour-related measurements (i.e., perimeter, area, centroid, etc.). It should be noted, however, that while the operational mechanics of the two low-cost distance sensors utilized in this investigation have been previously elucidated, their installation serves the purpose of collecting additional height measurements pertaining to the scene, rather than solely focusing on the height of the plants and so forth. To analyze measurements like lettuce’s diameter or area using physical measurement units such as millimeters, it is essential to perform the conversion of pixels to that desired unit. Despite the possibility of obtaining a reliable estimation of the pixel/mm scale in our experimental scenarios, as already discussed in Section 2.1.3 and Section 2.2.4, while illustrated also in Figure A2 and Figure A3 of Appendix A, this is reserved for future endeavors, and all measurements of distance and area in this study are presented in pixels. While this approach does not provide absolute measurements in metric units, it enables the comparative analysis of plants, as they are observed using the same imaging system.

Programming for the data analysis, regression, and visualization of the results was performed using Matlab^TM, ver. 2018b software (The Mathworks, Inc., Natick, MA, USA). The Raspberry Pi 4 of our robotic camera system was running Raspbian GNU/Linux 10 (buster, kernel: 5.10.63), and its programming was performed using Python, release 3.7.3, and OpenCV 4.5.5.

3.1. Evaluation of the Proposed Robotic System Setup and Its Movement’s Accuracy and Repeatability

Based on the information provided in Section 2.2.3 (also depicted in the flowchart shown in Figure 10), once each plant is successfully identified, a circumcircle is drawn around it to encompass the plant with the smallest possible area (minimum enclosing circle). This circle serves as a boundary for the entire plant, defining the extent of its leaves. Additionally, it is used to align the 2D projected image of the plant, which is obtained by cropping the RGB image. The position of the carriage, measured in steps, is then recorded in a file along with other relevant datalogging parameters. These recorded positions of all the plants detected beneath the aluminum beam act as photographic targets for further analysis.

The objective of this experiment was to assess the potential inaccuracies and misalignments in the construction of the system, as well as its ability to consistently perform the same task (repeatability). Additionally, the experiment aimed to evaluate the error difference between the intended (set) target and the actual outcome (accuracy). In this context, the RoboPlantDect’s system repeatability, refers to the quantitative assessment of its movement at specific points, ensuring consistency across multiple attempts. On the other hand, accuracy pertains to the system’s successful detection and localization of the same target (plant) during each iteration. A Mitutoyo 500-181-30 caliper (Mitutoyo Co., Kawasaki, Japan) with a measurement precision of 0.01 mm was utilized to conduct the manual measurements.

Table 2. Calculated distance from each of the 15 targets set and for 10 iterations.

Target	Steps	Absolute Distance from “Home” in X-Axis (mm)	Calculated Distance from the Target’s Center (mm): Red Laser
			Experiment’s Iteration										Mean	STD
			1	2	3	4	5	6	7	8	9	10
1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
2	1667	100.2	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0
3	5000	300	0	0	0	0	0	0	0	0	0	0	0	0
4	6500	390	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0
5	11,000	660	1	1	0.5	1	0.5	1	0.5	1	0.5	1	0.8	0.26
6	15,000	900	1.5	1.5	1.5	1.5	1	1.5	1.5	1.5	1.5	1.5	1.45	0.16
7	19,000	1140	1.5	1.5	2	2	2	1.5	1.5	1.5	1.5	1.5	1.65	0.24
8	22,000	1320	1	1.5	1	1.5	1	1	1	1	1	1	1.1	0.21
9	26,000	1560	1	1	1	1	1	1	1	1	1	1	1	0
10	30,000	1800	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0
11	33,000	1980	0.5	0	0.5	0	0	0.5	0.5	0.5	0	0.5	0.3	0.26
12	38,000	2280	0	0	0	0	0	0	0	0	0	0	0	0
13	43,000	2580	1.5	1.5	1.5	1.5	1.5	1.5	1.5	1.5	1.5	1.5	1.5	0
14	46,000	2760	1	1.5	1	1.5	1.5	1.5	1.5	1.5	1	1.5	1.35	0.24
15	50,100	3000	1	1	1	1	1	1	1	1	1	1	1	0

displays the calculated distances from the center of the target for each iteration of the experiments.

Based on Table 2 above, the mean (accuracy) for all iterations for the movement to target No. 6, located at 15,000 steps from the “Home” position, is 1.45 mm, while the standard deviation (repeatability) for the same point is 0.16 mm. Similarly, the mean (accuracy) for all iterations for the movement to target No. 7, located at 19,000 steps from the “Home” position, is 1.65 mm; the standard deviation (repeatability) for the same point is 0.24 mm. The results for these two specific positions but also at 43,000 and 46,000 steps (targets No. 13 and 14), where the mean was calculated to be 1.5 mm and 1.35 mm, respectively (standard deviations of 0 and 0.24 mm correspondingly), are the worst four observations regarding the system’s position accuracy and repeatability for all the experiments conducted. All the other targets were approached with an accuracy estimated in the range 0 to 1.1 mm and with a standard deviation in the range 0 to 0.21 mm. We conclude that the overall system’s positioning accuracy along the X-axis (the axis of linear motion) can be considered in the range 0 to 1.65 mm and its repeatability from 0 to 0.26 mm, respectively. The results presented in Table 2 are also visually depicted in Figure 14a.

In

Table 3. Measured deviation (system’s offset) on the Y-axis, recorded after the experiments that were conducted in various positions covering the full range of linear movement.

Target	Steps	Absolute Distance from “Home” in X-Axis (mm)	Measured Deviation along Y-Axis (mm): Red and Green Lasers
			Experiment’s Iteration										Mean	STD
			1	2	3	4	5	6	7	8	9	10
1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
2	1667	100.2	0	0	0	0	0	0	0	0	0	0	0	0
3	5000	300	0	0	0	0	0	0	0	0	0	0	0	0
4	6500	390	0	0	0	0	0	0	0	0	0	0	0	0
5	11,000	660	0	0	0	0	0	0	0	0	0	0	0	0
6	15,000	900	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0
7	19,000	1140	1	1	1	1	0.5	1	0.5	1	1	1	0.9	0.21
8	22,000	1320	1	1	1	1	1	1	1	1	1	1	1	0
9	26,000	1560	1	1	1	1	1	1	1	1	1	1	1	0
10	30,000	1800	1	1	1	1	1	1	1	1	1	1	1	0
11	33,000	1980	1	1	1	1	1	1	1	1	1	1	1	0
12	38,000	2280	1	1	1	0.5	1	1	1	1	0.5	1	0.9	0.21
13	43,000	2580	0	0.5	0	0.5	0	0.5	0	0	0	0	0.15	0.24
14	46,000	2760	0	0	0	0	0	0	0	0	0	0	0	0
15	50,100	3000	0	0	0	0	0	0	0	0	0	0	0	0

, the measured deviations along the Y-axis through the entire experiment are presented. The recorded outcomes are also visually depicted in Figure 14b. Based on the aforementioned table and the relevant plot, the mean displacement for all iterations when moving towards targets Nos. 7 and 12, as well as the intermediate targets, is approximately 1 mm. This indicates a standard deviation along the Y-axis within this range, which can be attributed to minor misalignments in the junctions of the three V-Slot 2060 aluminum beams that are connected together.

Despite securely attaching the beams on both sides with tee nuts, it was not possible to completely eliminate this offset. A potential solution would involve replacing the three aluminum beams, measuring 1.5 m and 1 m in length, with a single piece of aluminum beam measuring a total length of 3.5 m. However, this suggestion is reserved for a future update, as the one-piece beam was not available at the time of implementation. Furthermore, since this observation pertains to a standard offset, it can be compensated for through software means.

Nevertheless, the primary objective of the proposed linear robotic camera system is to monitor plant growth through photography. While it is desirable to achieve a high accuracy in the movement of the carriage, there is no requirement for sub-millimeter precision as would be necessary in medical interventions, for example. We believe that procedures and applications of this nature can be approached with a certain level of tolerance.

3.2. Experimental Results Example on Plant’s Identification and Localization

Figure 15 illustrates, as an example, the process of identifying and localizing six baby lettuce seedlings. The experiment consisted of 15 iterations, during which the pots containing the six plants, positioned at random intervals from each other, remained stationary.

In each iteration, the system was programmed to initiate the entire process of identification and localization (without any prior knowledge) every 10 min. Each row in Figure 15, labeled with the corresponding plant’s name (plant A to F), displays the basic sequence of plant identification, as depicted in the flowcharts in Figure 10 and Figure 11. Figure 15 column a showcases the six lettuce seedlings photographed in a single pass of the experimental procedure, encompassing all 15 iterations. The columns b and c on the same Figure 15 present the intermediate results of the morphological operations described in the Identify Plant procedure, as outlined in the flowchart of the Plant Finding procedure in Figure 10. Column d of Figure 15 displays the final outcomes of each plant contour found, represented in blue, along with their respective centroid, which is denoted by a small cyan circle. Additionally, the estimated minimum enclosing circle, plotted with each center, is depicted in red. In column e of the same Figure 15, the manually traced plant contours are shown in yellow. Each figure in this column also includes the respective centroid overlaid as a small blue circle, allowing for a direct visual comparison.

Table 4. Calculated mean and standard deviation for each plant localization (in steps).

Plant No.	Plant Localization Steps (Mean/Std)
A	$2624\pm 23.8$
B	$6833\pm 10.3$
C	$\mathrm{17,428}\pm 32.3$
D	$\mathrm{27,092}\pm 10.3$
E	$\mathrm{42,294}\pm 17.2$
F	$\mathrm{47,277}\pm 28.6$

summarizes the calculated mean and standard deviation of the carriage’s position (in steps) for each of the six lettuce’s (plant A to F) detection and localization task. All locations reported are with reference to the home position. Based on the measurements conducted, it has been determined that in order for the carriage to travel a distance of 3 m along the X-axis, a total of 50,100 steps from the stepper motor are necessary.

Consequently, it can be deduced that the carriage moves at a rate of 1 mm for every 16.7 stepper motor’s steps. According to the information presented in Table 4, the standard deviation (repeatability) for plant A localization in metric units is determined to be 1.43 mm. In the case of locating the target plants B to F, the standard deviations for their localization are as follows: 0.62 mm, 1.94 mm, 0.62 mm, 1 mm, and 1.7 mm, respectively.

Table 5. Evaluation metrics results for the 15 iterations experiment on 6 baby lettuces.

Evaluation Metric	Result
IoU	$0.761\pm 0.054$
Precision	$0.764\pm 0.046$
Recall	$0.979\pm 0.042$
F1 score	$0.857\pm 0.042$

presents the findings related to the calculated total projected leaf area through experimentation for the 15 iterations involving six baby lettuces, as compared to the area deduced from the manual tracing on the same images.

The evaluation metrics IoU, Precision, Recall, and F1 score are utilized to compare these outcomes with the corresponding plant features obtained through manual tracing of the lettuces’ contours.

During the whole run, the RGB and NIR cameras capture images with a resolution of 1920 × 1080 pixels (original images). As mentioned previously the RGB camera is utilized for the plant identification and localization task. Nevertheless, since the calibration parameters are known, a process can be conducted to correct lens distortions in both cameras. Additionally, a rectification process can be applied to the pair of stereo images, enabling the computation of a disparity map for depth estimations and 3D reconstruction or stitching alignment procedures. However, this latter aspect is reserved for future investigations. At present, the identification of plants is performed using the Identify Plant function (Figure 10), which is an ongoing iterative procedure aimed at successfully recognizing different plants. To enhance the efficiency of the image processing process for plant identification, only the RGB images are undistorted prior to being cropped. The color cropped images, which were further analyzed and processed in the specific experiment, had dimensions of 360 × 360 pixels. All images were saved in a PNG format and organized automatically into dedicated folders for each iteration. At the present time, the system logs the original images of both cameras together with all the intermediate (morphologically processed) images until the final, undistorted, cropped colored image outcomes, with the plant centered and circled. Furthermore, the system automatically logged and stored plant growth-related data measurements, along with the datalogging information, both locally and remotely. On average, the entire process, starting from the initial image acquisition at the home position to the identification and localization of the six baby lettuce seedlings, as well as saving all the relevant results, took approximately 6.5 min per iteration. Moreover, the system successfully identified and localized each plant within an average time frame of 55 s.

3.3. Measurements Related to Morphological Characteristics of the Plants

Figure 16 illustrates a detection and localization example of five lettuce plants over a four-week period. The identified plants, labeled as A to E, were consistently photographed every Thursday starting at 12:00 am, throughout the laboratory experiments.

The colored cropped images, which were further analyzed and processed in the specific experiment, had dimensions of 888 × 888 pixels. The red circle in the figures represents the minimum enclosing circle of each plant’s contour, which is plotted in blue. Additionally, the small red circle and blue square in each figure indicate the center of the circle and the centroid, respectively. The plot clearly demonstrates the successful detection and localization of all five lettuce plants across different growth stages.

In Figure 17, an enlarged depiction of plant A is provided, illustrating example results of the 4-week growth interval experiment. The experimental outcomes for this plant are presented in the first row of the figure, which aligns with the first column of Figure 16. The second row of Figure 17 displays the corresponding results of manually traced plant contours (represented by yellow squares) for the same images. Additionally, the centroid of each contour is overlaid as a small magenta circle, facilitating a direct visual comparison.

Table 6. Evaluation metrics results along with some growth-related parameters for the 4-week experiment.

Day	Plant No.	IoU	Precision	Recall	F1 Score	${\mathit{m}}_{\mathit{d}\mathit{t}}$	${\mathit{c}}_{\mathit{d}\mathit{t}}$	${\mathit{D}}_{\mathit{m}\mathit{a}\mathit{n}}$	${\mathit{D}}_{\mathit{e}\mathit{x}\mathit{p}}$	${\mathit{S}}_{\mathit{m}\mathit{a}\mathit{n}}$	${\mathit{S}}_{\mathit{e}\mathit{x}\mathit{p}}$	${\mathit{C}}_{\mathit{m}\mathit{a}\mathit{n}}$	${\mathit{C}}_{\mathit{e}\mathit{x}\mathit{p}}$
Day 10	A	0.952	0.997	0.955	0.975	0.87	0.37	215	217	0.42	0.46	0.79	0.80
	B	0.932	0.998	0.933	0.965	0.67	4.1	195	198	0.39	0.47	0.82	0.83
	C	0.943	0.995	0.947	0.971	0.90	1	231	231	0.33	0.38	0.79	0.80
	D	0.919	0.994	0.925	0.958	0.74	4.9	210	211	0.33	0.46	0.81	0.82
	E	0.939	0.996	0.943	0.968	0.43	1.44	235	239	0.31	0.39	0.75	0.77
Day 17	A	0.977	0.992	0.982	0.987	1.75	2.65	373	374	0.38	0.45	0.88	0.88
	B	0.974	0.998	0.977	0.987	0.82	0.99	375	376	0.41	0.51	0.85	0.86
	C	0.967	0.996	0.971	0.983	1.65	3.1	414	416	0.38	0.46	0.84	0.85
	D	0.962	0.998	0.964	0.981	1.66	2.37	379	387	0.36	0.45	0.88	0.89
	E	0.976	0.994	0.982	0.988	1.55	2.24	421	420	0.39	0.53	0.86	0.87
Day 24	A	0.991	0.995	0.996	0.996	0.67	4.6	632	631	0.63	0.61	0.90	0.90
	B	0.992	0.997	0.995	0.996	0.55	2.57	612	611	0.62	0.62	0.93	0.94
	C	0.991	0.996	0.995	0.996	0.69	0.21	565	565	0.62	0.64	0.94	0.94
	D	0.992	0.998	0.994	0.996	0.64	1.1	668	667	0.59	0.57	0.91	0.91
	E	0.990	0.997	0.993	0.995	1.1	2.3	593	594	0.63	0.62	0.92	0.93
Day 31	A	0.988	0.989	0.999	0.994	0.59	0.91	668	666	0.53	0.52	0.89	0.89
	B	0.989	0.991	0.998	0.995	0.98	0.12	640	638	0.65	0.64	0.93	0.93
	C	0.990	0.991	0.999	0.995	0.28	2.23	608	606	0.54	0.54	0.91	0.91
	D	0.991	0.993	0.998	0.996	0.38	8.32	681	678	0.58	0.59	0.90	0.90
	E	0.988	0.991	0.997	0.994	0.58	1.06	631	629	0.49	0.49	0.91	0.91

also displays, as an example, the results concerning the computed total projected leaf area as related to the manual traced outcomes for this 4-week experiment. The evaluation metrics IoU, Precision, Recall, and F1 score are utilized again to compare these outcomes with the corresponding plant features obtained through the manual tracing of the lettuces’ contours.

Table 6 also depicts the Euclidean distance $m_{dt}$ (in pixels) between the centroids of the polygonal shape contour, which is automatically computed experimentally for each plant, as related to the corresponding one calculated from the manual traced plant contours. In a similar fashion, the $c_{dt}$ (in pixels) is reported for the Euclidean distance of the centers of the minimum enclosing circles for each plant.

With the symbols $D_{man}$ and $D_{exp}$, the diameters (in pixels) are considered for these circles, which, at the same time, correspond to the maximal distance between two pixels belonging to the lettuce (by means of the longest distance between any two points on its contour’s perimeter outline). Other parameters frequently used for quantifying plant growth using image analysis, which are reported in Table 6 for the specific experiment, are the stockiness $S_{man}$ and $S_{exp}$, which is a measure of circularity, or in other words, the roundness, with a value in the range 0 to 1 and compactness $C_{man}$ and $C_{exp}$, which expresses the ratio between the area of the plant and the area enclosed by the convex hull (by means of the contour of the smallest convex polygon that surrounds the plant) [30,56,57,58,59,60]. The stockiness involves the area (A) and the perimeter (P) calculations of the leaves and depicts the difference with a circle. It is calculated as follows: $S=\frac{4·\pi ·A}{P^2}.$ In a similar fashion, the compactness is defined as the total leaf area divided by the convex hull area. The subscripts “man” and “exp” are used in the previously referred parameters to indicate the manual calculation and experimentally (automatically) computed values, respectively.

Table 7. Average results (mean and standard deviation) of the evaluation metrics and the growth-related parameters listed in Table 6.

Evaluation Metric	Result
IoU	$0.972\pm 0.023$
Precision	$0.995\pm 0.003$
Recall	$0.977\pm 0.024$
F1 score	$0.986\pm 0.012$
$m_{dt}$	$0.875\pm 0.445$
$c_{dt}$	$2.329\pm 1.977$
D	$1.900\pm 1.744$
S	$0.048\pm 0.045$
C	$0.006\pm 0.006$

summarizes the averages (mean and standard deviation of the outcomes) of all metrics and measured parameters depicted in Table 6 for the 4-week experiment. Specifically, for the case of the diameter ($D_{man}$ and $D_{exp}$), stockiness $(S_{man}$ and $S_{exp}$), and compactness ($C_{man}$ and $C_{exp}$) parameters, the mean and the standard deviation of the absolute difference between the manual and the experimentally calculated values are reported with the symbols D, S, and C, respectively.

In Figure 18, as an illustrated example, some of the lettuce plants’ parameters that were measured over time (in days) for the 4-week experiments are presented. The measurements were conducted manually and automatically for all five plants labeled A to E, with the data collected once per week. The parameters of the lettuce plants shown in the figure include the total projected leaf area (Figure 18a,b), the maximum diameter (Figure 18c,d), the compactness (Figure 18e,f), and the stockiness (Figure 18g,h) for both manual and automatic calculations in each instance.

In Figure 19, the detailed visual representation of the detected and ground truth total projected leaf areas overlaid, for all the plants in the 4-week experiment, is presented. The areas’ plots were created as solid polygonal regions defined by 2D vertices. These vertices correspond to each lettuce’s total projected leaf area contour, either manually traced or found experimentally.

In the first 2 rows (Days 10 and 17), the detected (experimental) area is clearly visible with a different color (lighter shade of pink), so as to emphasize the difference in the detection of the plant by our algorithmic procedure. In the last two weeks (Days 24 and 31), the corresponding overlaid, experimentally delineated area of the plant is almost identical to the ground truth and cannot be visually differentiated. This fact is also justified by the assessment criteria outlined in Table 6 for the specified experimental period, which indicate high confidence scores for IoU, Precision, Recall, and F1-score metrics.

This can be attributed to the laboratory experimental conditions under which the images were acquired, e.g., the mosaic pattern in the background floor but especially the organic fertilizer used to facilitate lettuce growth. This latter fact is proved from the results after processing the first two weeks’ images (Days 10 and 17), also affecting the detection accuracy. On Days 24 and 31, where lettuces are fully grown and cover the pot, the detection results for the leaf contours are almost the same as the ground truth images, as is clearly depicted from the last two rows of Figure 19. Both contours are almost identical for each plant. Another factor that can affect the segmentation result is the presence of gravel in the soil. To create a consistent soil backdrop in the images, one possible solution is to apply a thin layer of white sand or another monochrome growing medium, which helps in reducing the issue at hand. Nevertheless, this was not our intention. Lastly, Figure 20 depicts both the manual and the automatically calculated areas for each plant A to E, for the same experiment, together with the percentage (%) detection error for each case. With reference to this plot, the minimum error was calculated to be $0.031\%,$ while the maximum error was $7.51\%$.

To summarize, the outcomes of both tests are considered adequate for the realization of plant detection and localization in the context of plant photography. Moreover, the mechanical structural margin of error detected in the system’s accuracy and repeatability evaluation test is viewed as satisfactory, and we are confident that this degree of inaccuracies will not impede the operational efficiency of our system in practical, real-world situations. Furthermore, the experiments revealed a consistently smooth linear motion of the carriage. This observation confirms a well-designed experimental system that is properly assembled, which minimizes the risk of missing steps from the chosen stepper motor (i.e., the motor possesses adequate torque for the load). It also highlights a precise selection of initialization parameters for the stepper motor driver in the programming (maximum speed = 500 out of 1300, acceleration = 500 out of 20,000, microstepping resolution = 2). Also, the installed LEDs were successful in providing adequate illumination for the plants during image capture throughout the entire process. Furthermore, the inclusion of limit switches beneath the carriage not only improves the current level of precision but also adds an extra layer of safety at the far ends of the beam. This ensures that the “home” position is accurately identified before every initialization startup process. Considering the test outcomes and the overall system performance, it has been ascertained that no corrective actions need to be taken, apart from periodic maintenance, which is essential for all mechanical systems with moving components (e.g., fastening of the timing belt, leveling, inspection of the alignment of the carriage’s wheels to ensure their proper positioning, etc.). The attained degree of precision in motion is considered adequate for the intended purpose of capturing plant images for research experimentation, which is the main purpose of the current application. Additionally, the inclusion of the “Autofind” feature in our system provides an extra advantage (explained thoroughly in Section 2.2.4). In cases where the user’s initial plant number set for identification is ultimately unsuccessful (e.g., less plants were identified), this feature activates the LED strip to flash, notifying the user of the problem. In this case, the user has the ability to reset the alarm using the graphical menu and carry out preventive actions. The plant features collected by the automated linear robotic camera system proposed in this work, not only provide the potential for individual plant monitoring, but also facilitate the comparison of growth patterns across various environments.

3.4. Limitations, Future Outlooks, and Related Studies

Although the proposed system in this work has been shown to be accurate and efficient for its dedicated task tailored to plant photography for research purposes, there are still limitations that we need to consider. One limitation is that the images can only be acquired from the top view, indicating that the error may increase in the scenario where the foliage of neighboring plants partially overlaps. Even though there was no specific requirement for such an arrangement, a common assumption scenario observed in many growth monitoring and phenotyping analysis systems is the spacing of planting beds or the arrangement of pots in a photographical scene, ensuring sufficient distance between them, as, for example, in [4,25]. In other words, and specifically for our scenario, the five (or six) pots in each row were randomly positioned with sufficient spacing to prevent any overlap of foliage, even when the plants reached their full growth. However, future research endeavors will encompass cases where mature plants exhibit partially overlapping foliage, as the adjacent leaves’ segmentation control presents a promising and challenging aspect for the successful identification of plants in a general scenario [3]. Furthermore, future studies will organize dedicated and expanded experiments on a broader field of research to further test our system and to continue to collect images captured in its permanent installation place, that is, the greenhouse. One further limitation is the existing system’s ability to recognize and manage only a single plant at a time in a sequential manner, rather than processing multiple plants simultaneously, as demonstrated in [6,16,17,18], for example. Nevertheless, despite its usefulness and potential time-saving benefits, no such limitation existed for the detection of numerous plants and time-related concerns during our design phase. However, since to simultaneously manage multiple plants seems a necessary augmentation, we are currently implementing an algorithmic pipeline towards lettuce plant detection and localization on a panorama image, which is created by combing multiple overlapping images together (image stitching), captured from our system.

The results obtained from the conducted experiments clearly indicate that our linear robotic camera is capable of effectively identifying mature plants. Nevertheless, there is room for improvement, which could be achieved by incorporating a higher resolution binocular field stereo system. This parallel adaptation would greatly facilitate the investigation procedures and enable the detection of even the most intricate contours within the projected leaf area, including those that are exceptionally small. The selection of the current stereo camera module was based on its synchronization capabilities, as well as its similarities in characteristics and features to both RGB and IR cameras, as mentioned earlier. In this research, the calibration of the stereo system was extensively discussed, and the IR images were efficiently captured, stored in dedicated folders, and synchronized along with the colored images and datalogging information. However, the primary focus was on the task of plant recognition and localization, which was accomplished using the RGB camera and simple image processing techniques. Consequently, both images representing the same scene are readily available and can be further processed using image registration techniques to derive the relevant Normalized Difference Vegetation Index (NDVI) of the plants, which provides valuable information about their health status. Nevertheless, the calculations associated with the NDVI are reserved for future endeavors.

Moreover, as an ongoing project, our experimental plans include the incorporation of different imaging modalities, namely hyperspectral imaging in parallel investigations of different yet interesting image acquisition angle views, other than the top- or side-view imaging modes [41]. This approach, along with the simultaneous utilization of instruments such as the Konica’s Minolta SPAD-502 m and other affordable alternatives already in-house developed [61], which are employed for assessing plant health by analyzing their relative chlorophyll content, will permit the methods presented here to be incorporated into a broader field of research for plant growth monitoring.

We are of the opinion that there is room for enhancing the user friendliness of this system and expanding its applicability to a wider range of users. Our objective is not to encompass every conceivable plant species, but rather to establish a system that can be easily expanded and adapted to meet evolving needs. To this end, we are currently developing an enhanced iteration of our system. Our ongoing efforts involve expanding our robotic camera positioning system to function as a Cartesian robot. This advancement will enable us to cover a larger area and grant the camera platform the freedom to be positioned at any point within the x–y coordinate system, specifically above the crop canopy.

The utilization of small-scale robotic machines or CNC-based mechanization techniques in agricultural operations, as demonstrated in similar published works, as for example, in [4,14,25,38], showcases the effectiveness of employing Cartesian robots to address the issue of precise movement and among others, to cover a wider area of plants in the investigation, a fact that possesses a design advantage over our current system’s implementation. However, this can be achieved either by programming the cameras to move in stable increments and to stop at specific locations to acquire images [14], or through the establishment of predetermined coordinates and predefined targets [25,38], or even manually operating them to stop at specific locations to acquire images [4]. The image capturing and processing device, for example, as presented in [4], although interesting in its implementation, is entirely manually operated by a user. This user must utilize a toggle switch each time to maneuver the system forward or backward along the track for image data collection and the subsequent crop status monitoring. Our proposed system differs significantly from conventional manual practices, since it can operate autonomously, with minor user interaction. Furthermore, it features automatic identification, localization, and image capturing of the plants being analyzed, thus alleviating the burden of plant photography for human photographers and crop researchers. Moreover, in contrast to approaches that rely on fixed spaces or specific plant locations, as compared to the solutions presented in [14,25,38], the linear robotic camera system developed in this study continuously scans a row of target plants and accurately detects and localizes the plants themselves using simple image processing techniques and OpenCV primitives. This system allows users the flexibility to position the targeted plants in any desired arrangement inside a row, with the robot seamlessly adjusting to the new configuration. This facilitates a more efficient utilization of the typically limited space within a small greenhouse.

4. Conclusions

The current investigation introduces a new, cost-effective, one-degree-of-freedom linear robotic camera system for automatic plant photography. The system is specifically designed for the automatic and systematic identification and localization of plants ordered in a row, in a complete random design, which can thus can readily facilitate the greenhouse’s plant growth monitoring procedures. The entire system was meticulously constructed using readily available materials, parts, and basic electronic components. The primary objective of this study was to offer comprehensive information about the hardware components used, the software pipeline, and the implementation details of the proposed system. A preliminary assessment of the experimental prototype, conducted on a small sample of lettuce plants primarily in laboratory settings, demonstrated encouraging outcomes in terms of plant identification and localization. Although the measurements presented were performed on one type of lettuce plant, we believe that our device can perform just as well with various kinds of plants and in similar applications with minor modifications. Among other things, our device will be useful for creating custom image datasets useful for conducting testing and experimentation, while developing computer vision algorithms.

The system was designed and developed by a team of engineers from various disciplines (electronic, computer, and agricultural engineers) to fulfill realistic needs for automatic plant growth monitoring and imaging in an educational greenhouse. The length and height dimensions of the whole construction were chosen so as to align with the crop rows in the greenhouse, ensuring seamless monitoring and imaging processes without disrupting the existing infrastructure. The agricultural engineers, apart from the description of the problem and the initial technical requirements, also closely followed the progress of the implementation, evaluating the intermediate results and intervening with timely observations in the design decisions that were taken. This collaborative approach was continued during the whole development process and extended to both hardware and software development, so a device corresponding to the initial requirements was produced. The same procedure was followed for the evaluation of the construction’s operation to monitor the growth of illustrative lettuce samples, feeding back the design and implementation phase with new data that were incorporated into the device. Several such cycles of agile development needed to take place before the device started to operate as required.

Such a device is in line with the requirements of Industry 4.0 for smart agriculture, known as Agriculture 4.0, as it can contribute to the automation and modernization of agricultural production. Agriculture 4.0 refers to the anticipated radical changes in agricultural production through the application of recent advances in information technology. Agriculture 4.0 is a term for the next big trends to impact this industry, including a greater focus on precision agriculture with the regular application of the Internet of Things, autonomous robots, big-data, and artificial intelligence to drive greater business efficiencies, make smarter planning decisions, and lower the environmental footprint. Labor-intensive tasks will be undertaken by autonomous ground robots or unmanned aerial vehicles equipped with machine vision systems and precision telemetry. With Agriculture 4.0, the whole agri-food chain will become more competitive and profitable, as deeper insights and precise advice for plant requirements will empower farmers to minimize agricultural input (water, nutrients, plant protection, etc.), thus minimizing environmental and health hazards, but also to scale production further by reducing space requirements, growing periods, and logistic chains [62]. Devices such as the one presented here are a perfect fit towards this direction for the future, Internet-based agriculture, as it can provide farmers and agronomists with valuable remote monitoring of crop production and decision support, so critical decisions for crop growth and health are made on time. Our future plans include the in vivo evaluation of our prototype in the greenhouse, with various kinds of plants, to assure its performance in real conditions. The next milestone includes the generalization of this prototype to the two and the three dimensions, so it can cover larger surfaces and vertical farming, respectively, with almost the same robotic head, eliminating the need for one such device per row in a greenhouse.