Ultrasonic devices are based on the measurement of reflected sound waves. The estimation of the distance is based on the physical principle of time of flight, producing a short bust of sound in a unique direction. The wave returns to the receiver after impacting an object. The device measures the travel time of the acoustic signal and transforms it into a voltage signal. The output voltage can be converted into a distance. All readings were taken with a Pepperl+Fuchs (Mannheim, Germany) ultrasonic sensor (UC2000-30GM-IUR2-V15), pointing straight downward to the ground. This sensor is able to work in outdoor conditions since it has an internal temperature correction to compensate for the effect of temperature fluctuation on the electronics, a suitable degree of physical protection and a high resistance to vibrations. The ultrasonic sensor measured the distance to the crop and weed mixture covering the ground. The transducer ultrasound frequency is approximately 180 kHz with a sensor resolution of 0.48 mm when working in full evaluation range. The divergence angle was configured to result in a 0.20 m diameter footprint when placed at a height of 0.80 m (Figure 1). The response delay was used according to factory settings with a value of 195 ms. The sensing range was configured from 350 to 1,000 mm, which means that echoes generated out of this range will be signalled as erroneus measurements by the device (maximum voltage output value). A successful measurement needs a minimum surface to identify an object in the measument range.

The measurement accuracy of the device was assessed in a calibration step, where measurements over the whole range were taken and compared with the distances measured with a tape measure. The calibration equation with an R² of 0.99 was established for height determination, converting the voltage signal v [0–10 V] into a distance d [cm]: d = 7.0275v + 29.658.

The sensor was connected to a 12 V battery power supply. The output was measured with a data acquisition (DAQ) module Labjack U12 (LabJack corporation, Lakewood, CO, USA) connected via USB connector to a laptop. A software was developed for ROS (Robot Operating System [21]) to acquire time stamped raw sensor (voltage) outputs and distances with the calibration equation. The heights of the crop and weeds were estimated by subtracting the actual estimated distance from the reference distance (0.80 m). This distance was acquired during an initial system calibration step. In a weed free area distance readings during 10 s were averaged to measure the height of the sensor over ground. This distance was stored as reference distance for the working conditions. The vehicle was stopped at the sample locations, and samples were taken from the continuous measurements, labelled with the infestation type and stored for the analysis.

Field experiments were conducted in the south west of Germany at Ihinger Hof Research Station (Renningen, Germany) of the University of Hohenheim during 2012 in a 3.5 ha winter wheat field. The experimental station Ihinger Hof (48°7″N, 8°9″E; altitude 450 m) is characterized by a mean annual precipitation slightly higher than 700 mm and a mean temperature of 9 °C. The soil of the experimental field was loam. Winter wheat was sown with 17 cm of row spacing and fertilizers were applied at planting time. A pre-emergence herbicide treatment with glyphosate was applied before sowing. Readings were taken at two dates, the first assessment was carried out on 26 and 27 March, the second on 9 April in order to cover different weed and crop stages. Weeds were assessed when the crop growth was at the stages 11 to 13 and 15 to 17 of the BBCH scale [22]. The field was mainly infested with Echinocloa crus-galli (L.) P. Beauv., Lamium purpureum L., Galium aparine L., and Veronica persica Poir. Weed growth stages ranged from BBCH 9 to BBCH 15–16 on the first sampling date and BBCH 9 to BBCH 23–24 on the second date. A total of 80 sampling points were recorded on the first date and 40 on the second date. The locations were chosen to reflect different weed compositions of grass and broad-leaved weeds as well as mixtures of both, looking for an equitable distribution of pure samples of the present species and different mixtures compositions. At these locations the sensor was pointed towards the centre of the sampled area for height determination. Immediately after taking ultrasonic readings, weed and crop densities were counted manually and the heights of the weed groups (grasses, broad-leaved weeds) and crop were determined using a metric rule.

Additionally, an RGB (red, green, blue) image was taken for weed and crop coverage determination. The images were acquired pointing the camera downwards from 1 m height. A Nikon digital camera EOS 5D (Canon Inc, Tokyo, Japan) with 6.1 megapixel DX Format CCD image sensor equipped with an 18–70 mm lens was used to capture the digital images. Then, all weeds present in a 0.20 m diameter circle, coinciding with the sensor footprint, were separately hand-harvested by groups without disturbing the ground, and taken to the laboratory for biomass determination (dry weight). The measurements were repeated after weeding: the distance was measured and a second RGB image was taken for the evaluation of the crop coverage. Finally, the crop plants were also harvested for dry weight determination.

An image sample as taken in the field is illustrated in Figure 2 (left). The area of measurement was defined using a grey circular frame, which was visible in the images. To determine the region of interest (ROI) inside the ring, an image processing chain was developed. The first step was to segment the pixels belonging to the frame based on its colour. A transformation from the RGB to the HSV (hue, saturation, value) colour space was the first step, then each of the channels was thresholded separately with pre-defined intervals (H: [116–255], S: [0–128], V: [31–255], in 8 bit channels with a value range of [0–255]). This led to three binary images with pixels falling into the selected HSV value ranges marked black (value 0, denoting object), and pixels off the range marked white (value 255, denoting background). These three binary images were combined with a binary AND operator to a single binary image (Figure 2 right). Figure 2 (centre) shows a colour combination of the three thresholded channels: the black parts denote the object pixels, any other colour is background, its colour being out of the selected range in at least one thresholded HSV channel. Small noise objects remained, these were to a large extent filtered out with a median operator of size 10, since the frame appeared as linear structure with a width of about 10–20 pixels in the image. Small regions (having a small diameter), were filtered out with this approach (Figure 2 right).

From this segmentation the region of interest defined by the inner area of the frame could be selected in the next step. Since the frame can only be segmented partially due to plant overlaps and illumination changes (shadows), the circularity of it was chosen to be a robust feature. To find a circle that fits the segmented pixels of the frame, a circular Hough transform was applied to the binary image ([23] Section 5.4.3). The Hough transform is known to be robust against noise and missing object parts (here: parts of the ring). The circular Hough transform marks all possible locations of a circle centre for a foreground pixel position in a new image, which is a representation of the Hough space, also known as the accumulator space. The Hough space was defined on a grid of the same size as the image. The circle centre candidates are arranged in a circle around the foreground (frame) pixel positions according to the circle formula (x − x₀)² + (y − y₀)² = r². x₀, y₀ is the foreground pixel position and x, y are the possible locations of centres for a circle of radius r.

In the Hough space the value in these locations (circle) around a foreground pixel is increased by one. This process is repeated for each foreground pixel. If multiple pixels are in the radius distance from a circle centre, then they have a common centre location, and the count value of that location will be increased to the number of supporting pixels. Figure 3 (left) illustrates this process for three foreground pixels P1, P2 and P3, which are all located on a circle (circle with solid line). Construction of the possible centre locations (dotted circles) leads to intersections, which are possible centre locations: C12, C23, C13 are supported by two points, C123 by all three points and therefore the latter is the most probable centre location. In the discrete Hough space (Figure 3 top right), each pixel value can be interpreted as a probability value, and the pixel with the maximum value can be selected as most probable centre location. The ROI can be constructed as circle with the radius parameter of the Hough transform. The radius of the ROI was reduced by 10 pixels to avoid parts of the frame being within the ROI (Figure 3 bottom right). This region of interest was then used to mask the RGB image (Figure 4) and define the area for the further processing. It was defined in the alpha channel of the image. The alpha channel of an image is an optional, additional channel besides RGB and contains the opaqueness (resp. transparency) for each pixel. The background was set transparent and the region of measurement to opaque, as it can be seen in Figure 4 (the part out of the circular area is transparent).

The coverage was determined before and after removal of weeds and assessed with a second image processing approach. The first step of the ROI image processing was the calculation of the ExG (excess green index, ExG = 2G-R-B [24]), which leads to a grey level image with green objects (plants) appearing bright, in contrast to objects with a different colour, which appear dark. A threshold of 8% (corresponding to grey level of 20 = 255 × 0.08) was applied to the ExG, separating the objects pixel-wise into foreground (plants) and background in a binary image (Figure 4). From the binary, image the number of foreground pixels p_f and background pixels p_b were counted. Only pixels within the previously defined ROI were taken into consideration. The ratio C = p_f/(p_b + p_f) is then the percentage of plant coverage.

Correlation analysis was conducted to provide an initial examination of bivariate relationships among the variables. Multiple linear regression analysis was then used to explore the relationship between experimental parameters that explained the differences in ultrasonic heights for the crop-weed composition in the measured combinations. A set of twelve explanatory variables, including two categorical variables, was analysed. The two categorical variables were: weed presence with two levels, and weed infestation with four levels: non-infested, infested by grasses, infested by broad-leaved weeds and infested by mixtures. The ten continuous variables were: weed coverage, grasses height, broad-leaved weed height, crop height, grasses density, broad-leaved weed density, crop density, grasses biomass, broad-leaved weed biomass and crop biomass. Regression conditions for the variables were previously tested for linearity and zero mean error, and that the variance of the error is constant across observations (homoscedasticity). The predictors were linearly independent and the errors were uncorrelated and normally distributed. A canonical discriminant analysis (CDA) was used to classify and predict the weed compositions. This methodology is a pattern recognition approach to find an optimal linear combination of input features for the separation into groups based on the dependent variable. This method expresses the dependent variable as functions of linear combinations of the input features [25]. In this study, two cases were distinguished: (I) the capabilities of the method to separate infested areas from non-infested; and (II) the possibilities of discrimination of four types of infestation levels: (a) non-infested; (b) infested by grasses; (c) infested by broad-leaved weeds and (d) infested by both weed types. The analyses were independently performed for both sampling dates using the SPSS statistical software (v. 20, IBM Corporation).

High correlations between ultrasonic readings and the analysed variables for the first date were observed, these were significant on a level of α = 1%. Ultrasonic readings were well correlated with weed coverage reaching a Pearsons R² of 0.40. For weed density and weed biomass measurements these values were higher with an R² of 0.94 and 0.90 respectively. Figure 5 shows the simple regression for weed density and weed biomass with ultrasonic heights. The r² denotes the correlation coefficent of the simple regression with fixed intercept (computed with the lm funciton of R statistics software). Individually, ultrasonic readings showed the highest correlations with the measured parameters of the grasses, with R²= 0.62 for height, R²= 0.67 for density and R²= 0.62 for biomass measurements. On the second sampling date no significant correlations were found for any of the measured parameters.

The comparison of the input variables indicated good relationships to the height data obtained with the ultrasonic device, with an adjusted R² of 0.67 for the multiple regression model (first date). The actual height, density and biomass parameters of grass weeds had an influence in the model with a p-value >95% for the first two paramenters and a p-value >90% (Table 2). In the case of broad-leaved weeds, density was excluded from the model and actual height and biomass were significant on a level of 95% and 90%, respectively. The influence of the crop on the regression was not significant due to its constant height and biomass levels.

For the second date no significant result could be obtained. Therefore, weed detection is feasible only in early development stages of cereal crops with ultrasonic sensors.

The CDA was conducted for the two categorical variables. For the first sampling date, significant differences between infested and non-infested spots in the first sampling period were observed. Additionally, significant differences were found for separating infestation types of the four situations: non-infested, infested by grasses, infested by broad leaves and infested with both weeds. With a CDA satisfying results for the binary classification of the samples were achieved (Table 3). The situations were grouped into two classes (infested, non-infested). With these two groups the infested areas were correctly classified in 92.8% of the cases. Misclassified samples could be due to a later growth stage of the crop covering weeds. All non-infested cases were properly classified into the non-infested group. However, when weeds were classified into four groups, the results led to poorer results than for the binary classification, as shown in the confusion matrix in Table 4.

On the second sampling date, the use of CDA, as method for group classification, was not possible due to insignificant differences between groups. Weeds shorter than the crop were missed in cases, where the crop almost completely covered the weeds. No significant differences between groups in any of the situations were observed.

A previous study in maize showed the possibilities of the use of ultrasonic sensors for weed detection in the inter-row area [9]. In the case of maize, the crop did not interfere the sensor output, since the ultrasonic readings were taken in the crop-free inter-row area and the advanced growth stage of weeds slightly improved the classification results. In later growth stages the differences between weed groups were more prominent. For cereal crops the situation is different, since no crop-free areas can be measured. Due to the faster crop growth the crops cover the weeds earlier. Problems with detection occur towards later growth stages.

The effect of overlapping was dependent on the crop size, which was different for the two measurement dates. Crop plants were in both cases taller than the weeds and covered them, but the measurement principle of ultrasonic devices gives an explanation for the observed differences. The sensor generates a wave that is reflected as soon as it finds a minimum surface to create a reliable echo [26]. Since the ultrasonic measurement principle is dependent on the angle of the surface the signal hits, the minimum surface is dependent not only on the total leaf area, but also on the direction of the leaves. Plant species determination from ultrasonic signals has been explored by Harper and McKerrow (1999) [27]. In early stages, even when the crop is taller than weeds, the weeds can be measured, because the coverage by the crop is not reaching the minimum surface for echo generation. However, when the crop is in a later growth stage, the leaf coverage of the crop is large enough to reflect the ultrasonic wave and initiate an object measurement. Weeds, which were shorter than the crop, therefore did not influence the reflected echo in the second sampling. The values obtained from the second date illustrate the necessity of an early detection. A shorter, earlier time window to carry out the measurements has to be chosen, and this period of time coincides with the time window for weed treatments. Comparably, machine-vision techniques, which have shown its possibilities for ground detection of weed infestations [28], need to sample in a relatively small time window for successful weed identification. However, some advantages arise from the use of ultrasonic sensors.

The overlapping effect was reported to be a problem also for image based weed recognition approaches. In the case of image analysis, overlapping between weeds and crop make the classification more difficult [29–31], which represents a serious problem for the use of this methodology. In the case of ultrasonic measurements this effect is more dependent on the leaf size of the overlapping leaves, whereas in image processing already small leaves can lead to combined objects.

Even though ultrasonic measurements are unable to identify weed infestation separated by species, they can be used for the application of herbicides (or mixtures) with a broad spectrum. This approach was used for the development of a prototype automated variable rate sprayer for on-line application of agrochemicals in wild blueberries [32]. The researchers also concluded that the use of the ultrasonic measurements are accurate enough for patch spraying and suggested the possibility of detection at early stages, similarly to the situation found in cereal crops. Indeed, one of the major advantages of its use would be the possibility of its use for real time operation. Patch spraying can be controlled directly according to the measurements, without the need for an additional previous sampling and mapping step. The ultrasonic output is easy to process and the high number of readings per second lead to a high spatial sampling resolution in the direction of travel. Another advantage is the relatively low costs of these sensors that allow using multiple sensors in parallel. Thus, the whole field can be scanned with a high spatial resolution, which is necessary for a detailed application decision for patch spraying.

Compared with optoelectronic sensors, the obtained results of this study reveal a potential of this technique to additionally assess weed density, not only coverage [33]. Since the measurement principle is non-optical, ultrasonic sensors could be combined with optoelectronic sensors (sensor fusion, as proposed by Adamchuk et al.[34]) to obtain more reliable estimations especially for low infestations. The application window for on-field operations could be extended this way. No interferences between the two types of sensors are expected, since the measurement is based on different principles (light, sound). Additionally, the ultrasonic sensor principle is not affected by ambient light conditions, thus it can be applied during night hours without the need for advanced illumination. The costs for the sensors are lower than for optoelectronic systems, providing a budget solution for real-time applications to the end-users.