In this section, experiments on synthetic and real datasets are conducted to verify the proposed method with three references detailed in literatures [
16,
20,
24]. In order to decrease the influence on localization result due to different parameter settings, for both the proposed and referenced methods, each chessboard corner is detected with the same initial pixel coordinates, and refined from the same local neighborhood with a square size of 31 × 31 pixels.
4.1. Synthetic Data
In order to acquire synthetic chessboard image, a pin-hole camera is simulated with the properties: [
fx,
fy] = [7000, 7000], [
u0,
v0] = [1296, 972]. The image resolution is set to 2592 × 1944. A single chessboard pattern with 20 mm cell size in both directions is projected to the image plane. Since optical paths are reversible, an ideal projection from the pattern center can be found and defined as ground truth. Gaussian blur with the window parameter
σf and Gaussian noise with 0 mean and standard deviation
σn are added to make the image similar in appearance to a real one (
Figure 5). For each given
σf and
σn, 100 independent trials are performed, with other simulation parameters varied and limited in their ranges (
Table 1), under the premise of ensuring faultless projections.
Figure 6 depicts the RMS error of sub-pixel localization as a function of
σf and
σn. The proposed technique performs significantly better than the referenced ones. Although it results in a higher error due to the increase of
σf and
σn, the performance drop is not as pronounced as for the others. Concretely, for the poorest image quality (
σf = 3,
σn = 0.2), the result shows that the errors are about 0.154, 0.041, 0.077, and 0.024 pixels for [
16,
20,
24], and the proposed technique, respectively. Remarkably, Placht et al. [
24] yields a stable, but significant, error in the presence of the change of
σf and
σn for taking filtered images as inputs. That is to say, it not only eliminates noise distinctly, but also leads to an extra uncertainty of sub-pixel localization.
In addition, sub-pixel localization errors from all trials (the total number is 40,000) are gathered for an overall evaluation represented by boxplots. As shown in
Figure 7, for the proposed and referenced methods, interquartile ranges (IQRs) are highly symmetrical about medians pretty close to zero. In detail, the IQRs are about 0.18, 0.13, 0.32, and 0.04 pixels in both directions for [
16,
20,
24], and the proposed method, respectively. The smaller IQR reflects the better performance of sub-pixel localization. Again, using filtered images as inputs lead to a particular outcome, that there are no outliers to be distinguished with the largest IQR for [
24].
The above simulation relies on the assumption that edges defining a corner are completely straight in the observation area, or region of interest, where the corner is going to be found. However, it is well known that lenses inevitably have distortions. To obtain maximum allowable distortions for the method, another simulation is conducted, with the fixed blur strength and noise level (
σf = 1.5,
σn = 0.1), and the first order radial distortion with the degree
k1 is added to the image (
Figure 8). Again, for each given
k1, 100 independent trials are performed, with other simulation parameters varied and limited in their ranges (
Table 1), except for [
tx,
ty,
tz] set to [115, 80, 1000], for ensuring the projections farther away from the principal point.
Figure 9 depicts the RMS error of sub-pixel localization as a function of
k1. The highest errors are 0.089 pixels for [
16], 0.046 pixels for [
20], 0.123 pixels for [
24], and 0.037 pixels for the proposed method. Again, the proposed method performs significantly better than the referenced ones when
k1 varies from −5 to 5. Different from [
20] and the proposed method, Bok et al. and Placht et al. [
16,
24] show a distinct variability due to the limitation of their methodologies; the blur strength and noise level in the simulation have greater impact on the localization result than the distortion. For practical applications, however, cameras with the coefficient
k1 larger than 5 are lesser used in photogrammetry because the pinhole model is no longer applicable for them. Therefore, for calibrating a camera for common use, the proposed method can be effectively performed without any pretreatment.
4.2. Real Data
In contrast to simulations, real data experiments cannot directly evaluate the accuracy of sub-pixel localization via the observed corner coordinates, due to their undetermined ground truth data. An alternative and indirect way is examining it based on camera calibration technique.
Figure 10 shows that a camera (JPLY, G1GD05C) with 16 mm lens and 2592 × 1944 image resolution is employed for conducting a camera calibration experiment based on a coordinate measuring machine (CMM) (Brown & Sharpe, Global Image 7107) with a single chessboard pattern (20 × 20 mm cell size) mounted on the end of its probe. 3-D control points are achieved by programmatically driving the probe to a set of specially designed positions, and provided with a dimensional error of less than 0.003 mm in both directions. For each position, the chessboard pattern is recorded by the camera for capturing a corresponding corner. Since all corners are located at the sub-pixel level, the camera can be calibrated based on bundle adjustment [
4,
6].
Table 2 lists the result of intrinsic parameters calibrated from the corners based on four different approaches. According to the definition of radial distortion coefficients detailed in [
2], for [
16,
20,
24] and the proposed method, the maximum distortions evaluated using the image point furthest from the principal point are 22.78, 25.92, 29.33, and 24.54 pixels in the radial direction, respectively. Among them, the contributions of
k2 are 2.03 pixels for [
16], 6.06 pixels for [
20], 10.06 pixels for [
24], and 4.62 pixels for the proposed method. Therefore,
k2 has a much smaller influence on the pixel offsets than
k1. Or rather, the estimator of
k2 is more sensitive to noise in the corner coordinates. In spite of the fact that the result cannot intuitively demonstrate the performance of each approach, it is pivotal for the following investigations.
Figure 11 represents four scatter plots of re-projection errors. For general examinations, the maximum and mean re-projection errors for the proposed method are 0.22 pixels and 0.11 pixels, evidently less than 0.32 pixels and 0.15 pixels for [
16], 0.29 pixels and 0.15 pixels for [
20], 0.39 pixels and 0.18 pixels for [
24]. From the standpoint of addressing perspective-n-point problem, the re-projection errors, assessing the validity of calibration, are subjected to some optical indications, e.g., image and lens resolutions, and integrated with certain methodologies, including calibration model, target geometry, and sub-pixel localization. The mentioned experiment employs a robust model with stereo points establishing correspondences between world and image frames accurately and, therefore, the lower re-projection errors not only reflect the better solution of perspective-n-point, but also testify the higher accuracy of sub-pixel localization. Therefore, the corners obtained using the proposed technique are better suited for camera calibration.
In order to alternatively examine the proposed technique, different measurements on displacement and attitude are carried out using the CMM and camera mentioned above. Firstly, for displacement measurement, a target (6 × 6 grid of points, 20 × 20 mm cell size) fixed on the end of the probe is moved with the guide and imaged by the camera placed in front of the CMM, for measuring a distance
d between two different positions as an evaluation factor (
Figure 12). Secondly, for attitude measurement, two targets, T
1 and T
2, mounted on the base with the same grid and cell as that of the above measurement, are imaged by the camera (
Figure 13). Among the three axis vectors, only the one in the z direction can be perfectly measured using the probe (Renishaw, SP600), by scanning the pattern plane of each target, due to a restriction that makes it hard to capture 3-D coordinates of a corner accurately, by means of contact measurements. Thus, the included angle
θ between two normal vectors is adopted as another evaluation factor more suitably. Fifteen independent trials are performed to localize sub-pixel corners, employing both the proposed and referenced methods, and estimate the camera poses from their respective intrinsic parameters listed in
Table 2. The metrics
d and
θ are then computed for investigating discrepant deviations with respect to the CMM data.
Figure 14 presents the results from the above measurements. Under the premise that the CMM provides baselines with a higher accuracy, the RMS errors of
d and
θ are 0.032 mm and 0.010° for [
16], 0.021 mm and 0.009° for [
20], 0.037 mm and 0.013° for [
24], and 0.014 mm and 0.006° for the proposed approach. Although there are many estimable and inestimable influences during the experiments, the results are mainly dependent on the accuracies of intrinsic parameters and corner coordinates, and essentially subject to the performance of each sub-pixel localization method because the camera is also calibrated from the respective corner set. From a synthetical point of view, exact values of
d and
θ are derived from reliable estimations of camera poses predetermined by accurate corner coordinates. As an apparent outcome of the comparison, the proposed technique presents a higher performance than others.
As shown in
Figure 15, in order to test the proposed approach in terms of its robustness to real-world data gathering, four images of a stationary chessboard are captured by the mentioned camera under underexposed, overexposed, indoor light interfered, and outdoor light interfered scenarios. For each corner in a 6 × 6 array, its maximin deviation between different scenarios is computed and gathered for an overall evaluation.
Table 3 lists the overall evaluation result for four different approaches. The RMS deviations are 0.419 pixels for [
16], 0.287 pixels for [
20], 0.396 pixels for [
24], and 0.241 pixels for the proposed approach. Considering the fact that the relative pose between the target and camera is stationary, the variability of each detected corner is mainly subject to the robustness of corner localization in the presence of the ambient light changes. The smallest RMS deviation proves that the proposed approach has higher interference immunity, resulting from a more robust corner model.
4.3. Practical Application
The proposed approach is implemented in a visual measurement system called 3D four-wheel aligner (3Excel, T50). The system, designed for aligning four automobile wheels, mainly consists of an upper computer and four cameras and chessboard targets (
Figure 16). Each camera is equipped with infrared filter and illuminant, for ensuring a high immunity to the complicated imaging conditions at customer sites. During an initial operation, the automobile under test is driven up to a certain distance by external force. Meanwhile, the cameras C
1 to C
4 are triggered in synchronous mode to capture image sequences of the targets T
1 to T
4 mounted on the front-left, front-right, rear-left, and rear-right wheels, respectively. For each image sequence, sub-pixel corners are detected for estimating a wheel attitude with respect to the corresponding camera; two alignment parameters
toe-
in/
toe-
out and
camber are then determined by decomposing angles of the wheel attitude unified in a global frame defined by the bodywork. During a real-time alignment, the parameters are dynamically calculated from continuous estimations of the wheel attitude changes with respect to their initial values.
As demonstrated in
Figure 17, an automobile (Ford Focus) in healthy condition is used for on-site alignment. The introduced aligner can capture chessboard images with black backgrounds due to the usage of infrared filters and illuminants. After finishing the initial operation, the alignment is carried out and divided into two periods: one performs normally, and the other is interfered by infrared pollution sources. During each period, the alignment parameters are incessantly computed based on both the proposed and built-in techniques, until the number of their recorded values reaches 120.
Figure 18 shows two boxplots of total
toe-in/
toe-out for the front and rear wheel-sets. This parameter, called
toe-in for positive and
toe-out for negative values, is defined for investigating the symmetry of each wheel-set about the geometric centerline (or thrust line). For both normal and interfered periods, the proposed method results in a median closer to zero and IQR of minor scope, compared with the built-in algorithm. Considering the fact that two total values should be pretty small because of the healthy condition of the automobile, the boxplots prove that the proposed method shows better central tendency, due to the accurate corner localization. When comparing the medians of the proposed method during two periods, the deviations between them are about 0.003° and 0.002° for the front and rear wheel-sets, significantly less than that of the built-in algorithm (0.009° and 0.003°), which also shows that the proposed method has a higher interference immunity resulting from the self-checking technique.
Figure 19 depicts four curve plots of
camber as a function of time stamp for the front-left, front-right, rear-left, and rear-right wheel positions, which are divided into two parts, according to two different periods of the alignment. This parameter is defined for measuring the inclination of a wheel with respect to vertical line of the bodywork. Different from
toe-in/
toe-out, it is separately investigated using the wheel attitude, and weakly restricted to the absolute symmetry about its baseline for the corresponding wheel-set and, therefore, a total value makes poor sense for the evaluation. However, when observing the median change between two periods of each front wheel, there is a strong comparison that the difference is less than 0.003° for the proposed method, and more than 0.008° for the built-in algorithm.
It should be remarked that both methods yield median changes of the rear positions smaller than that of the front ones. This can be found from both
Figure 18 and
Figure 19, and especially for the built-in technique. There is a logical explanation, as follows: the distances from the front and rear wheels to the infrared pollution source are about 1.5 m and 3.9 m, respectively. The energy of interference is in a state of decay when the distances become larger and, therefore, has no pivotal influence on corner localization and pose estimation for the rear wheels. That is to say, when there is a lack of robust localization technique, a direct way to improve system accuracy is enhancing image quality. Or rather, perspective-n-point is prone to errors if there are outliers in the set of point correspondences. Thus, the self-checking technique can be used in conjunction with existing solutions to make the final solution for the camera pose more robust to outliers.