Color Point Defect Detection Method Based on Color Salient Features

Abstract

Display color point defect detection is an important link in the display quality inspection process. To improve the detection accuracy of color point defects, a color point defect detection method based on color salient features is proposed. Color point defects that conform to the perception of the human vision are used as the key point for detection. First, the human visual perception constraint coefficient is used to correct the RGB three-channel image to obtain the color-channel-transformed image. Then, the local contrast method is used to extract the point features of the color channel, which achieves point defect enhancement, noise and background suppression. Finally, the mean and standard deviation of the defect feature maps of R, G, and B channels are calculated. The maximum mean and standard deviation are selected as thresholds using the maximum fusion criterion to perform binarization segmentation of the defect feature maps of R, G, and B channels. An OR operation was performed on the segmented images and the point defect segmentation results were combined. The experimental results show that the average detection accuracy and recall of the algorithm is higher than 94%, which is a significant improvement compared with mainstream detection methods and meets the needs of industrial production.

1. Introduction

The development of information technology has increased the demand for displays. Especially with the advent of the 5G information era, smartphones, as terminal products, can no longer fully meet the needs of people’s production and life. The new generation of flexible displays and smart wearable devices bring new challenges and requirements to the development of display technology. This has led to an upgrade in the production of existing displays, in which machine vision technology is used to improve the productivity and reduce the production costs of factories [1]. The factory quality inspection of the displays is partially automated by automated optical inspection (AOI) equipment. However, the defect detection algorithm of AOI equipment limits its application. The detection of display defects in industrial production still relies on human eyes for detection, which hinders the production efficiency [2]. Therefore, developing a more effective defect detection algorithm to achieve machine vision instead of manual inspection is a pressing issue.

Display defect detection techniques have made great progress. The existing display defect detection methods are divided into three main categories: image registration-based methods [3,4,5], background reconstruction [6,7,8,9,10,11,12] and deep learning [13,14,15,16,17,18,19]. Zhu et al. [3] proposed the use of Fourier-Mellin Transform for the coarse registration of images, the use of accelerated robust features for precise registration of images, which improved the accuracy of the registration method. However, the image registration-based method cannot align images with simple backgrounds. Sun et al. [10] proposed a defect detection method using cascade mean shift and level set algorithm, where mean shift was used to detect defect candidate regions and solve the problem of the level set method being sensitive to contours. However, the method consumes a significant amount of time and cannot meet the demands of real-time detection. Lin et al. [13] proposed a deep channel attention-based classification network (DCAnet) as a defect image feature extractor, and proposed an adversarial training algorithm based on convolutional neural network (CNN) to achieve defect detection with sparse defect samples. Zhu et al. [14] proposed a hierarchical multifrequency-based channel attention network for effective scratch defects’ detection, which utilizes an attention mechanism to solve the problem of large variation in aspect ratio in defect samples. Zhu et al. [15] proposed a YoloV3 [20]-based defect detection scheme deployed in an AOI device for display point defects, which can detect defects in multiple backgrounds simultaneously. The deep learning approach requires the construction of a large amount of sample data to achieve effective detection, and thus has poor performance with small amounts of sample data.

Color point defects are common defects in the powered-on display. The size of the point defect is not fixed, while the color and contrast of the point defect will also change as the display changes. Therefore, this paper mainly addresses two problems: (1) the low accuracy of point defect detection caused by the contrast variation of point defects; (2) the problem of high false detection rate of the existing algorithm detection when the number of point defects is large. The contrast of a point defect is determined by brightness and color. The main goal of the existing conversion between color and grayscale images is to represent the contrast information between different colors in the original color image with grayscale information as best as possible. However, the implementation effect does not follow the human eye’s perception of color for conversion. Therefore, the converted grayscale image significantly differs from the original color image as perceived by the human eye, and its use in defect detection can lead to a high false alarm rate. The research on the shape features of point defects focuses on the detection of small targets, which provides a new way of thinking for the implementation of point defect detection.

We propose a color point defect detection method for displays based on color-salient features by analyzing the image characteristics of color point defects. For the morphological characteristics of point defects, the local contrast method was used to calculate the features of point defects and obtain the contrast feature maps in the three color channels that conform to the point morphology, achieving point defect enhancement, noise and background suppression. For the color and contrast problems of color point defects, the color point defects that match the human vision perception were used as the key point for detection. The human vision perception constraint coefficients were used to correct the three RGB channels and obtain the color-channel-transformed images. The mean values and standard deviations of the contrast feature maps of the three color channels were calculated separately. The maximum mean and standard deviation were selected as thresholds using the maximum fusion criterion, and the adaptive segmentation thresholds were selected to perform a binarization segmentation of the defect feature maps with three color channels. The ‘or’ operation was applied to combine the point defect segmentation results of the three channels and obtain the color point defect results.

2. Related Works

2.1. Display Weak Contrast Defect Detection Method

In recent years, TFT-LCD display defect detection technology has rapidly developed, with the low-contrast defect detection method becoming a research hotspot. Ngo et al. [6] proposed using multiple background reconstruction methods to obtain a differential map to solve the problem of low-contrast defects’ detection. Yang et al. [7] proposed a two-stage defect detection method that uses polynomial background reconstruction to mark anomalous regions as candidate regions for defects, and then uses a gradient-based level set method to accurately segment the defects in the candidate regions, which is effective for the detection of large, low-contrast defect regions. Jin et al. [21] proposed a two-segment exponential transformation method to achieve the effective separation of defects from the background and used Otsu’s method to achieve the accurate segmentation of defects. Cui et al. [22] proposed using Otsu’s method for defect candidate region selection under various display screens, followed by the variance and grid division of the original image, for defect detection. The above methods require the assumption that the detection target cannot have both low-contrast and high-contrast defects, which means they are ineffective in solving the point defect detection problem.

To find methods capable of detecting point targets with different contrasts, we were inspired by the detection of small targets in infrared. The characteristics of infrared small targets are more consistent with point defects. Various small object detection algorithms have been proposed in the field of infrared small-object detection [23,24,25]. Inspired by human vision mechanisms, Chen et al. [26] proposed a local contrast-based calculation method that can effectively detect small objects in bright spots. However, it is not as effective with background noise suppression. Wei et al. [27] proposed a multiscale patch-based contrast measure (MPCM) method for the above problems, which uses the center surrounding characteristic of the human eye, can effectively detect small targets in infrared rays (IR) images and can suppress background noise. However, such methods mainly focus on small infrared objects and do not use color information, so the detection of color point defects is not effective.

2.2. Multi-Scale Local Contrast Detection Method

The MPCM [27] method is a classical algorithm for infrared small object detection. Since the morphological features of the infrared small target are consistent with the display point defect features, we adopted the local contrast and multi-scale feature fusion calculation method proposed in the MPCM method.

First, a sliding window is used for local contrast detection on the input image. The sliding window is divided into two parts: the center point T and the background, where the background is further divided into eight parts. The definition between the center point and the background is as follows:

$$D\left(T\right)=\left(\begin{array}{c}d\left(T,B_1\right)\\ d\left(T,B_2\right)\\ \dots \\ d\left(T,B_8\right)\end{array}\right),$$

(1)

$$d\left(T,B_i\right)=m_T-m_{B_i},$$

(2)

where $d\left(T,B_i\right)$ represents the degree of difference between the target and the background, $m_T$ represents the pixel mean value of the target area, and $m_{B_i}$ represents the pixel mean value of the background area. Based on Equation (2), the measurement method of local contrast at a single scale is given. The calculation method of local contrast is as follows:

$${\tilde{d}}_i=d\left(T,B_i\right)*d\left(T,B_{i+4}\right),\left(i=1,2,3,4\right)\dots$$

(3)

$$S=\underset{\hspace{0.17em}\mathrm{i}\hspace{0.17em}\dot{}=1,2,3,4}{\min }{\tilde{d}}_i$$

(4)

where ${\tilde{d}}_i$ represents the gray level difference between the central region and the surrounding neighborhood in different directions, and S represents the single scale directional fusion feature map. When the local contrast in the four directions of the center position has a large value, the position is the target area and the target area is significantly enhanced. When the local contrast in the four directions of the center position is not consistent, the center position is noise interference and its value is calculated as negative.

A multi-scale feature map fusion needs to calculate the directional feature fusion maps of different scales, and then perform scale fusion according to the following equation. The scale fusion rules are as follows:

$$C\left(x_{ic},y_{ic}\right)=\underset{i=1,2,3\dots }{\max }{S}_i,$$

(5)

where $C\left(x_{ic},y_{ic}\right)$ represents the multi scale feature fusion map, and $S_i$ represents the multi scale directional fusion feature map.

3. Methodology

3.1. Algorithm Architecture

We propose an algorithm architecture for color point defect detection based on color-salient features, as shown in Figure 1.

As shown in Figure 1, our method consisted of three parts. The first part was the visual perceptual conversion correction of color channel images to achieve the representation of luminance information to color information. The second part was multi-scale local contrast feature fusion to realize the extraction of point defect features. The third part was the fusion feature segmentation to realize the color point defect segmentation results that match the human eye’s perception.

3.2. Color Channel Transformation

The goal of the detection of colored point defects was to achieve point defect detection that conforms to the human perception. When the RGB image was converted into a single-channel grayscale image for detection, the color feature of the point was converted into a mixed color and brightness feature, which led to detection results in a grayscale image against the visual perception of the human eye. Therefore, to restore the perceptual characteristics of the color image as the human eye in the gray channel, we performed color channel transformation, and the calculation method was described as:

$$\left\{\begin{array}{c}{R^{\prime }}_{gray\left(x,y\right)}=w_1\times R\left(x,y\right),\\ {G^{\prime }}_{gray\left(x,y\right)}=w_2\times G\left(x,y\right),\\ {B^{\prime }}_{gray\left(x,y\right)}=w_3\times B\left(x,y\right),\end{array}\right.$$

(6)

where ${R^{\prime }}_{gray\left(x,y\right)}$, ${G^{\prime }}_{gray\left(x,y\right)}$ and ${B^{\prime }}_{gray\left(x,y\right)}$ represent the image after R, G, B channel transformation. The $w_1$ represents the constraint coefficient of human visual perception of red, $w_2$ represents the constraint coefficient of human visual perception of green, and $w_3$ represents the constraint coefficient of human visual perception of blue.

3.3. Multi-Scale Local Contrast Feature Fusion

For the morphological features of point defects, we used the local contrast calculation method proposed by [27] for the feature extraction of point defects.

The transformed images of the three obtained color channels were mean filtered using templates of different scales, and the mean filtering results of the three channels were obtained. The mean-filtered images were computed according to Equations (1)–(3) for the local contrast.

The eight directional feature maps under single-pass and single-scale were fused into four directional composite feature maps. The four directional composite feature maps were fused according to the corresponding point minima to obtain the directional feature map fusion under single scale, and the fusion criterion was performed with Equation (4).

For multi-scale feature map fusion, the directional fused feature maps under three scales were calculated separately. Then, the scale fusion was performed using the scale maximum fusion criterion in the form of (5) to achieve the scale feature map fusion under a single channel. The multiscale feature fusion map of R, G and B channels were calculated.

3.4. Adaptive Threshold Map Segmentation

The statistical features of the local contrast feature maps of the three channels were analyzed, and their means and standard deviation were calculated using the following equation:

$$\mu =\frac{1}{M\times N}{\displaystyle \sum }_{i=1}^M{\displaystyle \sum }_{j=1}^N{C}_{i,j},$$

(7)

$$\delta =\sqrt{\frac{1}{M\times N}{\displaystyle \sum }_{i=1}^M{\displaystyle \sum }_{j=1}^N{(C_{i,j}-\mu )}^2},$$

(8)

where $\mu$ represents the mean value and $\delta$ represents the standard deviation of the multi-scale feature fusion map computed by the three channels R, G and B, while $\delta$ and $C_{i,j}$ represent the multi-scale feature fusion map.

As shown in Figure 2, there was a large difference in the mean and standard deviation of the obtained feature maps for the three channels. This is because the target region was enhanced and the background region was suppressed in the feature maps. We used the maximum feature fusion method to fuse the features of the image. The fusion rules were as follows:

$${\mu }_{max}=\max \left({\mu }_i\right)=\left\{\begin{array}{c}{\mu }_r,i=1,\\ {\mu }_g,i=2,\\ {\mu }_b,i=3,\end{array}\right.$$

(9)

$${\delta }_{max}=\max \left({\delta }_i\right)=\left\{\begin{array}{c}{\delta }_r,i=1,\\ {\delta }_g,i=2,\\ {\delta }_b,i=3,\end{array}\right.$$

(10)

where ${\mu }_{max}$ represents the maximum mean value obtained by fusion, and ${\delta }_{max}$ represents the result obtained by fusion of the maximum standard deviation.

Therefore, the threshold segmentation in this paper was implemented by the following formula:

$$th={\mu }_{max}+K{\delta }_{max}$$

(11)

The mean and standard deviation calculated for the three channels retain the maximum value as the standard for segmentation, where $K$ is the empirical value. Finally, the three-channel segmented images were ORed to obtain the defect detection result.

4. Experimental Results

4.1. Experimental Setup

To verify the point defect detection capability of the proposed method in multiple backgrounds, we used grayscale transition, cross test and solid color (including red, green and blue) backgrounds for evaluation. The defect-free test screen image, shown in Figure 3, was 687 × 1107 and had 100 images of each type. All images had corresponding ground-truth (GT) maps, in which defect areas were set to 1 and non-defective areas were set to 0. All experiments were run under MATLABR2018b with Intel Core I7-7700CPU@3, 60 GHz, 16 G RAM and Windows 7 64-bit operating system.

4.2. Algorithm Parameter Design

In our proposed method, it was necessary to determine the human eye’ perception coefficient of the color channel and the number of scales. Therefore, we experimentally compared the effects of different parameters on our detection results.

TDR is defined as the sum of correctly detected pixels in the test image divided by the sum of true cross-defect pixels, and FDR is defined as the ratio of falsely detected pixels to total detected pixels. We used TDR and FDR metrics for the algorithms in this paper to evaluate the defect detection results of our method.

$$\mathrm{TDR}=\frac{\left|S\cap GT\right|}{\left|S\right|},$$

(12)

$$\mathrm{FDR}=\frac{\left|S\cap GT\right|}{\left|GT\right|},$$

(13)

where S is the threshold segmentation binary image, and GT is the ground truth image.

4.2.1. Constraint Coefficient Optimization

The coefficients for human eye perception in color-to-grayscale images were proposed to be 0.299, 0.587 and 0.114. However, this is the perception coefficients of reflected light in natural scenes, which is different from the human eye perception of LCD autoluminescence. Therefore, we conducted experiments on the above coefficients to obtain the following results.

First, we fixed the coefficient of B channel to 0.114, and kept the sum of the coefficients of the three channels as 1, while changing the coefficients of the R channel and the G channel to obtain the defect detection values as follows.

Figure 4 shows that as the R-channel coefficient increases, the TDR reaches its peak (92.86%) and then decreases rapidly, and the inflection point of the FDR (100%) also appears in the peak part of the TDR, where the parameters of the peak are 0.499, 0.387 and 0.114, respectively.

We fixed the coefficient of the R channel as 0.299, kept the sum of the coefficients of the three channels as 1 and changed the coefficients of the G and B channels to obtain the following defect detection values.

Figure 5 shows that as the G-channel coefficient increased, the TDR reached its peak and then decreased rapidly and the peak of the FDR also appeared at the same time as TDR. The peak TDR and FDR values were 91.83% and 100% respectively, where the coefficients at the peak were 0.299, 0.387 and 0.314.

The coefficient of the G channel was set to 0.587, while keeping the sum of the coefficients of the three channels as 1, and the coefficients of the rest channels were varied to obtain the following values of defect detection.

Figure 6 shows that as the B-channel coefficient increased, the accuracy of the TDR decreased somewhat, roughly between 40% and 60%, but the recall was always 100%, which indicates that changing the value of the B-channel would result in a larger false detection rate.

Through the above experiments, we determined the optimal parameters to be 0.499 for the red channel, 0.387 for the green channel and 0.114 for the blue channel, which helped achieve maximum accuracy and recall.

4.2.2. Scale Parameter Design

The number of layers of local contrast was related to the size of the defect, and we needed to set different scales to accurately detect point defects. Because we determined the optimal number of stacked layers by analyzing the defect detection accuracy, the dimensions of the layers were set as shown in

Table 1. Contrast Layer Dimensions.

Layer	$A_1$	$A_2$	$A_3$	$A_4$
Size	$3\times$3	$5\times$5	$7\times$7	$9\times 9$

.

The calculated TDR and FDR results of different sizes and combinations are shown in

Table 2. Detection results for layer combinations.

Combinations	TDR (%)	FDR (%)
${\mathrm{A}}_1$	0	0
${\mathrm{A}}_1{\mathrm{A}}_2$	57.14	100
${\mathrm{A}}_1{\mathrm{A}}_2{\mathrm{A}}_3$	92.86	100
${\mathrm{A}}_1{\mathrm{A}}_2{\mathrm{A}}_3{\mathrm{A}}_4$	92.85	100
${\mathrm{A}}_2$	92.85	18.57
${\mathrm{A}}_2{\mathrm{A}}_3$	100	15.21
${\mathrm{A}}_2{\mathrm{A}}_3{\mathrm{A}}_4$	100	15.38
${\mathrm{A}}_3$	100	19.44
${\mathrm{A}}_3{\mathrm{A}}_4$	100	16.86
${\mathrm{A}}_4$	100	15.73

.

Table 2 shows the detection ability of different layers, where the recall rate is lower when the size is larger. The fused features of different layers effectively reduced the false detection rate and improved the accuracy rate. The fusion manner ${\mathrm{A}}_1{\mathrm{A}}_2{\mathrm{A}}_3$ achieved the best defect detection results.

4.3. Result Analysis

All original images in Figure 7 are defective displays collected in actual production. In the GT image and the result image, black indicates no defects and white indicates defects. Figure 7 demonstrates the point defect detection ability of the proposed algorithm. The point defects included mainly single bright, dark and colored points against a solid-color background, while the number of point defects greatly increased against a gray background.

As shown in Figure 7, our method could accurately detect the dot defects on solid-color background with a small number of dots. One of the dot features was salient in the contrast map of the corresponding channel, which shows that the contrast detection algorithm was effective. On gray backgrounds, multicolor point defects with different contrast levels were also detected, and the detection results matched human vision well.

As shown in

Table 3. Metrics of our proposed method.

	TDR (%)	FDR (%)
Single point defect	100	100
Multi-point defects	94.6	98.1

, we evaluated the detection ability of our proposed algorithm with quantitative metrics. For single-point defect detection, 100% accurate detection can be achieved, and the algorithm reaches 94% for multi-point defect detection.

4.4. Comparison of Different Methods

We compared the following algorithms with ours: method 1 with a polynomial-based background reconstruction [12], method 2 with a background reconstruction based on discrete cosine transform (DCT) [21] and MPCM [27] for comparative experiments on multi-color point detection.

As shown in Figure 8, both the DCT-based background reconstruction method and the polynomial fitting method were unable to detect the point defects against such a background. Both DCT and polynomial fitting methods were not able to obtain defect-free feature maps and effectively separate background and defect information, among which the polynomial fitting method was not effective at suppressing the information in the edge part of the cross-test image. To summarize, the performance of such methods is poor for the detection of color point defects.

As shown in

Table 4. Comparative evaluation of different methods.

Detection Method	TDR (%)	FDR (%)
DCT	35.7	26.8
Polynomial Fitting	19.6	19.7
MPCM	90.2	69.8
Ours	94.6	98.1

, the MPCM method has a considerable ability to detect point defects, which overcomes the interference of brightness uneven background and other noises. However, it mainly targets the detection of small infrared targets and does not use the color perception information as the human vision, so its recall rate was low. Our proposed method introduces the information on human vision perception of color and uses a unified segmentation criterion with fused features, so that the detected color dot defects conform to the detection results of human eye perception and achieve a high accuracy rate.

5. Conclusions

We propose a color point defect detection method based on color-salient features. The use of color features for point defects is achieved by introducing color channel constraint coefficients that match the visual perception of the human eye. The extraction of point features and enhancement of point defect information are achieved using a multi-scale local contrast approach. The effective detection of color point defects is achieved using a feature fusion segmentation method. Our method can detect point defects against a variety of display backgrounds. For a single point defect, our accuracy is close to 100%. For multiple point defects, the accuracy can still reach more than 94% and the detection results can be consistent with human eye perception. In future work, we need to improve the accuracy of multi-point defects.