Automatic Defect Detection for Web Offset Printing Based on Machine Vision

Abstract

In the printing industry, defect detection is of crucial importance for ensuring the quality of printed matter. However, rarely has research been conducted for web offset printing. In this paper, we propose an automatic defect detection method for web offset printing, which consists of determining first row of captured images, image registration and defect detection. Determining the first row of captured images is a particular problem of web offset printing, which has not been studied before. To solve this problem, a fast computational algorithm based on image projection is given, which can convert 2D image searching into 1D feature matching. For image registration, a shape context descriptor is constructed by considering the shape concave-convex feature, which can effectively reduce the dimension of features compared with the traditional image registration method. To tolerate the position difference and brightness deviation between the detected image and the reference image, a modified image subtraction is proposed for defect detection. The experimental results demonstrate the effectiveness of the proposed method.

1. Introduction

Automatic surface defect detection based on machine vision has been widely used in many industry fields, such as the semiconductor manufacturing industry [1,2,3], electronic industry [4], textile industry [5] and printing industry [6]. Especially in the printing industry, defect detection is indispensable to ensure the quality of the printed product. The traditional defect detection method still depends mainly on human visual inspection, which has many problems including slow speed, high cost and subjective instability. As a result, there is high demand for developing an automatic defect detection system based on machine vision in the printing industry.

Several defect detection methods have been proposed for web offset printing [7], printed text [8] and variable data prints [9]. Although there are many different kinds of defects in printed products, such as white spots, hickeys, streaks, color defects, etc. [10], from a technical point of view, the existing defect detection methods consist of the following major parts: reference image selection, image registration of the reference image and print image, subtraction of the detected image with the reference image for defect indicating.

For the selection of reference image, defect-free printing samples are first collected, and then are averaged as the reference image. Besides the reference image, two tolerance images are built with respect to the reference image for giving the maximum and minimum deviations of a pixel value [7,11]. However, how to build tolerance images adaptively is still an open research issue.

Due to mechanical vibration and distortion of the material of the printed matter, there is some position offset between the defective image and the reference image. Before doing image subtraction, the defective image has to be registered with the reference image. For image registration, the most common method is to select a special region of interest (ROI) and then use a correlation technique to match the detected image and the reference image [10,12,13]. However, searching the ROI is time-consuming, and the matching method based on correlation is sensitive to changes of light and noise. Considering limited rotate variance, multiple templates were set up in reference [13] for rotate deviation. The drawbacks of this method is that it is not able to adapt to large angle rotation.

After the defective image and the reference image have been registered, the image subtraction method is the most common method for identifying defect pixels. In reference [14], a method combining grayscale and gradient differences was proposed, where the grayscale difference between the reference image and the detected image was performed to determine the defect in the non-edge region while the gradient difference was used to determine the edge defect. Katsuyuki et al. [10] proposed an index space method to inspect printing defects. To improve the detection precision, a dynamic threshold with a hierarchical detection strategy was proposed in reference [12]. To avoid the contour artifact, a two-way image subtraction algorithm was proposed in reference [15]. In addition to the image subtraction, the incremental principal component analysis was used to detect printing defects for the first time [16].

Currently, deep learning has been successfully used in various fields [17,18]. It is gaining more and more attention of researchers in the field of defect detection [19]. However, labeling defect regions is time-consuming, and there are only a few defect samples in practice [20], which limits the application of the deep learning method in the field of printing defect detection.

For defect detection of web offset printing, research has rarely been conducted on this topic except reference [6,7]. The study in reference [6,7] is ground-breaking work. However, there are still some problems not investigated in reference [6,7] and much room for improvement. (1) Due to the continuous movement of printed paper in web offset printing, and the time uncertainty of beginning image collection, how to determine the first row of captured images is a new problem not studied before. For this issue, a fast computational algorithm is proposed by converting 2D image matching into 1D feature matching. To further speed up the search process, the pyramid algorithm is employed in the proposed algorithm. (2) Nowadays, most image registration methods rely on ROI-based or feature point-based matching methods, which are sensitive to light changes and noise. Unlike the existing methods, a shape context descriptor based on the concave-convex feature is presented for image registration, which can effectively reduce the dimension of features compared with the traditional image registration method [21]. (3) For defect identification using the image subtraction method, false defects occur easily because of a position difference and a brightness deviation between the detected image and the reference image. To overcome this problem, the traditional image subtraction method is modified for defect detection, which can tolerate the position and brightness differences in a limited scope, and thus can reduce the false detection rate.

The paper is organized as follows. Section 2 gives the methodology, which includes the algorithm for determining the first row for web offset images, the proposed image registration method based on the shape concave-convex context descriptor, and the modified subtraction method. The experimental results and discussion are shown in Section 3. The conclusion is drawn in Section 4.

2. Methodology

The proposed defect detection method consists of two parts as shown in Figure 1, the first part is template construction, while the second part is defect detection for printed images. For the first part, multiple defect-free samples are collected, and then are aligned to generate template images which include the reference image, a bright image and a dark image. The reference image is obtained by averaging multiple aligned defect-free images, and is used to register the detected image during the process of defect detection. In addition, two tolerance images, named bright image and dark image, are built with respect to the reference image. The bright image and the dark image give the maximum and minimum deviations of a pixel value, and in combination with the reference image serve as template images for further defect detection. For defect detection in the second part, the web offset image is first cut to the same size as the reference image by the algorithm in Section 2.1, then aligned with the reference image and compared with the bright image and the dark image. If a pixel value of the detected image is not in the range of the bright image and dark image, it is identified as a defect pixel. Finally, a Blob analysis algorithm is used to label defect region.

2.1. Determining First Row in Web Offset Images

2.1.1. Description of the Problem

For web offset printing, the printed paper roll is moving continuously through many color printing units which are used to transfer images from a printing plate cylinder to print media. Similarly, printed images are captured by line scan CCD cameras with the movement of printed matter. However, the height of a reference image is equal to the perimeter of printing plate cylinder, this raises a problem of determining the first row in captured images. From the first row of captured images, the subsequent detected image can be repeatedly obtained by cutting the captured images as the same rows of a reference image. Normally the camera works in conjunction with a rotary encoder mounted on the final printing cylinder as shown in Figure 2a, where the zero position of the rotary encoder can be used to determine the first row. The drawbacks of mounting a rotary encoder on the printing cylinder are the inconvenient installation and it can easily cause the instability of printing cylinder. To solve this problem, a new solution where the rotary encoder is installed in the position of the winding unit is given in Figure 2b. In Figure 2b, the cylinder mounted rotary encoder can be individually manufactured, is easy to install on the winding unit and in particular, will not affect the final printing unit. Thus, the solution in Figure 2b is a universal solution. Since the cylinder mounted rotary encoder on the winding unit is not in synchronization with the printing cylinder, how to determine the first row of captured images is a new problem as shown in Figure 3. Figure 3a is a continuous captured printed image, in which the red arrow indicates an image with a printing plate cylinder perimeter. Figure 3b is a reference image cropped from the captured image with a length of one printing cylinder perimeter, while Figure 3c is an initial frame image captured at the start time with the same size as the reference image during the detection process. In Figure 3c, the position marked by the red arrow may be the first row that corresponds to the reference image. Thus, how to quickly find the first row in the initial frame image is addressed in this section.

2.1.2. Search Method Based on Image Projection

Generally, the common method for determining first row of web offset images is the method of template matching based on normalized cross-correlation (NCC). For the template matching method, a sub-image with a discriminative feature from reference image should first be selected, and then search the corresponding region in the initial frame image. However, searching for the position of a sub-image in a large image is computationally expensive [1]. Moreover, selecting a sub-image with a discriminative feature is not an easy job if done manually, and in some cases a sub-image with a discriminative feature does not exist in the reference image. In this subsection, a fast searching algorithm based on image projection is given, which can convert 2D searching into 1D searching using NCC.

Inspired by the idea of searching for the first row only in the row direction, a 2D image can be converted into a 1D feature descriptor by projecting a 2D image along the row’s horizontal direction, and then the search process can be executed in 1D space. Figure 4 gives two gray average projection curves of Figure 3b,c. From Figure 4, we can see the projection curves of a reference image and an initial frame are similar except having a shift in the horizontal direction, which means the first row can be found when the two projection curves overlap simply by moving one of the projection curves. Thus determining first row in web offset images becomes a matching problem in 1D feature space. Considering the periodicity of the captured image, the projection curve of an initial frame is extended by a period as shown in Figure 5, where the projection curve of the reference image is matched with the extended projection curve of the initial frame image.

Suppose the reference image and initial frame are represented as $R\left(m,n\right)$ and $F\left(m,n\right)$ with the size of $w*h$, where $w$ is the image width and $h$ is the image height. The projection average gray values of $R\left(m,n\right)$ and $F\left(m,n\right)$ along with row direction are denoted by $r\left(n\right)$ and $f\left(n\right)$, which are given in Equations (1) and (2).

$$r(n)=\frac{1}{w}{\displaystyle \sum _{m=1}^{w}R(m,n)}n=1,2,\cdots ,h$$

(1)

$$f(n)=\frac{1}{w}{\displaystyle \sum _{m=1}^{w}F(m,n)}n=1,2,\cdots ,h$$

(2)

For convenience, we rewrite $r\left(n\right)$ as $t={\left(t_i\right)}_{i=1}^h=\left(r\left(1\right),r\left(2\right),\cdots ,r\left(h\right)\right)$, the extend $f\left(n\right)$ as $d={\left(d_i\right)}_{i=1}^{2h}=\left(f\left(1\right),f\left(2\right),\cdots ,f\left(h\right),f\left(1\right),f\left(2\right),\cdots ,f\left(h\right)\right)$. Then, NCC is employed for searching the first row as follows.

$$NCC(r)={\displaystyle \sum _{i=1}^h\frac{(t_i-\overline{t})(d_{i+r}-{\overline{d}}_r)}{\sqrt{{\displaystyle \sum _{i=1}^h{(t_i-\overline{t})}^2}{\displaystyle \sum _{i=1}^h{(d_{i+r}-{\overline{d}}_r)}^2}}}}$$

(3)

where $\overline{t}$ and ${\overline{d}}_r$ are the means of $t$ and $d$ and range from $d_{r+1}$ to $d_{r+h}$, respectively, which are given by:

$$\overline{t}=\frac{1}{h}{\displaystyle \sum _{i=1}^h{t}_i}\mathrm{and}{\overline{d}}_r=\frac{1}{h}{\displaystyle \sum _{i=1}^h{d}_{i+r}}$$

To speed up the search process, two strategies are imported into the proposed algorithm. The first is to select some columns which have obviously changing data as the projected ROI, as shown in Figure 6 marked with blue color, for reducing projection time. While the other is to construct a pyramid with one-dimension as shown in Figure 7, where searching process goes from the top layer to the first layer. The process of constructing the pyramid and searching first row is as follows:

(1)

Smooth the initial projection average gray value by three-point average and set it as the first layer of the pyramid.

(2)

Construct next layer by down-sampling the current layer with a sampling rate of $n$. In our experiment, $n$ is set as 2.

(3)

Repeat Step 2 until the number of the current layer is no less than 32, where 32 is an empirical value.

(4)

Search from the top layer to first layer using Equation (3). Suppose the best position index in the current layer is $i$, then the searching range in the next layer is $i\pm n$.

(5)

The best position index in the first layer is the first row.

2.2. Image Registration Based on Shape Context with Concave-Convex Feature

2.2.1. Introduction of Shape Context

Image registration plays an important role for defect detection based on image subtraction, as it enables the detected image to be in the same position as the reference image for comparison. In the process of image registration, common features shared by the detected image and reference image, such as feature point, region or shape, should be extracted first. Among these features, shape is the most popular feature for image registration, which has invariant characteristics for light changing. Thus, shape descriptor is explored in this paper for image registration.

Since shapes are represented by a series of contour points, shape description can be expressed by the relative position of contour points. In 2002, Belongie et al. [21] proposed a shape context descriptor to describe this relation. The main idea of shape context is to create a feature descriptor for each contour point relative to other contour points. As shown in Figure 8, for any point, a series of circles centered on that point were set up, and these circles were divided into different sectors, then the feature descriptor of this point was obtained by counting the number of contour points in each sub-region with a different circle radius and sector. For two shape matching, any point in a shape should be matched with all points in another shape by computing a cost function, which is computationally expensive.

2.2.2. Shape Context Based on Concave-Convex Feature

Inspired by shape context, a simple yet effective image registration method based on the shape’s concave-convex feature is presented. As shown in Figure 9, a rectangular coordinate system with the shape center is set up, and then a descriptor is given for each point, finally the shape descriptor is obtained by counting the distribution of all point descriptors.

For a point $p_i$ as shown in Figure 8, its descriptor is defined as ${\lambda }_i=(d_i,q_i,{\theta }_i)$, where $d_i$ is the distance from $p_i$ to $\mathrm{o}$, $q_i$ is the quadrant number of $p_i$, and ${\theta }_i$ represents the shape concave-convex feature of $p_i$ relative to $p_{i-1}$ and $p_{i+1}$. It is worth noting that our proposed shape concave-convex feature only relies on its two adjacent points, not all points used in Figure 7. For distance $d_i$, it is quantized to a trisection according to the maximum distance $d_{\max }$. The value of $q_i$ ranges from 1 to 4. The range of ${\theta }_i$ is between 0 and $\pi$, and is uniformly quantized to six segments. Considering all combinations of $d_i$, $q_i$ and ${\theta }_i$ values, there are 72 ($3\times 4\times 6$) different combinations. Thus, we build a histogram with 72 bins by counting all points on the shape, the histogram is used as the shape descriptor.

Let $f(p_i)=f(d_i,q_i,{\theta }_i)\in [1,72]$, then the histogram is constructed as follows.

$$h(k)=\{\#p_i|f(p_i)=k,i=1,2,\cdots ,N\}k=1,2,\cdots ,72$$

(4)

where $N$ is the number of points on the shape.

If there are two shapes, their corresponding histograms are $h_p$ and $h_q$, a similarity measure between two shapes is defined as follows:

$$S=\frac{1}{2}{\displaystyle \sum _{k=1}^{72}\frac{{[h_p(k)-h_q(k)]}^2}{h_p(k)+h_q(k)}}$$

(5)

In the process of print production, the distortion of a printed image can be approximately regarded as rigid distortion. Therefore, the transformation parameter can be evaluated by Equation (6).

$$\left(\begin{array}{c}\widehat{x}\\ \widehat{y}\end{array}\right)=\left(\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)+\left(\begin{array}{c}{t}_x\\ t_y\end{array}\right)$$

(6)

where $\theta$ is the angle of the rotation, $t_x$ and $t_y$ are the translation along the X-axis and Y-axis, respectively.

2.3. Defect Detection Based on Image Subtraction

As shown in Figure 1, the proposed defect detection algorithm consists of two parts: template construction and defect detection for a printed image. Unlike the traditional method, the constructed template images include a reference image, a bright image and a dark image. The traditional defect detection method is image subtraction, where the defective image is subtracted from the reference image, and then a fixed threshold is employed to determine whether a pixel is defective. However, using a fixed threshold may easily lead to false positive or false negative pixels, since the fluctuating range of intensity value is different with the pixel position. To solve this problem, two tolerance images, named bright image and dark image, are built with respect to the reference image. The bright image and the dark image give the maximum and minimum deviations of a pixel value, as well as in combination with the reference image serve as template images for further defect detection.

Let ${\mathrm{I}}_i(x,y)(i=1,2,\cdots ,n)$ denote defect-free samples, where $n$ is the number of defect-free samples. The reference image ${\mathrm{I}}_a(x,y)$ and the standard deviation $\sigma (x,y)$ of defect-free samples are expressed as Equations (7) and (8).

$${\mathrm{I}}_a(x,y)=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\mathrm{I}}_i(x,y)}$$

(7)

$$\sigma (x,y)=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^n{({\mathrm{I}}_i(x,y)-{\mathrm{I}}_a(x,y))}^2}}$$

(8)

The bright image ${\mathrm{T}}_b(x,y)$ and the dark image ${\mathrm{T}}_d(x,y)$ can be defined as follows.

$${\mathrm{T}}_b(x,y)={\mathrm{I}}_a(x,y)+\max (a,b\times \sigma (x,y))$$

(9)

$${\mathrm{T}}_d(x,y)={\mathrm{I}}_a(x,y)-\max (a,b\times \sigma (x,y))$$

(10)

where $a$ and $b$ are two empirical values that need to be determined by experimentation.

Besides the deviation of pixel values, there is still slight distortion of image contour because of changes to the material of printed matter and machinery shake, which cause false defects at the edge of patterns. To avoid this problem, operations of gray dilation and gray erosion with a $3\times 3$ structure element are implemented for the bright image and dark image, respectively. Figure 10 gives an illustration of the reference image, bright image and dark image.

For defect detection, the continuous collected web offset image is first cut to the same size as the reference image by the algorithm in Section 2.1, and then is aligned with the reference image. In the next process, the detected image is compared with the bright image and the dark image pixel-by-pixel. If a pixel value of the detected image is not in the range the bright image and dark image, it is identified as a defect pixel. Finally, a Blob analysis algorithm is used to label the defect region.

3. Experiment

In this section, the methods proposed in this paper were executed on a personal computer with an Intel core 3-2100 in VC++6.0, and two experiments were presented: (1) verifying the validity algorithm for determining first row in web offset images, (2) evaluating the performance of the proposed defect detection method.

3.1. Performance of Determining the First Row in Web Offset Images

For determining the first row in web offset images, two factors may affect the speed and precision, which are the number of layers in a pyramid and the size of the ROI. For the precision, it is defined as the number of rows searched by our algorithm deviated far from the actual row. In practical application, plus or minus one row is acceptable.

Table 1. Experimental results of different pyramid layers for 20 images.

Number of Pyramid Layers	1	2	3	4	5	6	7	8	9	10
Precision	±0	±0	±0	±0	±1	±2	±3	±6	±7	±10
Times/ms	3062.43	1293.86	591.27	318.24	185.46	126.46	106.71	85.18	80.30	80.98
Number of correct images	20	20	20	20	20	20	20	18	14	12

is the experimental results of different pyramid layers for 20 images with a size of $6200\times 8192$. As shown in Table 1, the searching speed becomes faster and faster with the increase of pyramid layers, while the precision is reduced. From the experimental results in Table 1, the conclusion is that a pyramid with five layers is the best choice, which achieves a balance between the time and the precision, in particular, the speed of five layers is approximately 20 times faster than the speed of a one-layer pyramid.

To speed up the searching process, some columns with the image height of the reference image were selected as the ROI, the number of columns is called the size of ROI.

Table 2. Executing time in different sizes of ROI (ms).

	Size of ROI	20	100	1000
Image Height
1164		1.60	2.05	7.08
8192		9.85	11.85	38.01

gives the searching times for 20 images with different image height and three different sizes of ROI. In Table 2, all 20 images are correctly matched with the reference image, and the pyramid with five layers is used. From Table 2, we can see the best size of ROI is 20, and the corresponding time is less than 10 ms, which can satisfy the requirement of real time in the printing industry.

3.2. Performance of the Proposed Defect Detection Method

In this subsection, two indices, called sensitivity and FPR (False Positive Rate), are employed to evaluate the performance of the defect detection method, which are defined as follows:

$$sensitivity=\frac{\#TP}{\#TP+\#FN}\times 100\%$$

(11)

$$FPR=\frac{\#FP}{\#TP+\#FN}\times 100\%$$

(12)

where TP represents defects that are correctly identified as defects. FN represents defects that are incorrectly identified as non-defects. FP represents non-defects that are incorrectly identified as defects.

The values of $a$ and $b$ in Equations (9) and (10) are two important parameters. If the parameter values are set too large, some defects will be incorrectly regarded as non-defects. In contrast, if they are set too small, some non-defects will be labeled as defects. In the experiment, 15 types of web offset images with different sizes were analyzed to determine parameters $a$ and $b$ in Equations (9) and (10). From the experiment, the best parameters are $a=15$ and $b=5$.

Table 3. Performance of the proposed defect detection performance. FPR, false positive rate.

Image Size	Number of Training Samples	Number of Defects	Number of Correctly Detected Defects	Number of Falsely Detected Defects	Sensitivity (%)	FPR (%)	Time (ms/Image)
4096 × 942	10	601	500	29	83.19	4.83	140.13
4096 × 942	20	601	549	18	91.35	2.30	140.13
4096 × 942	30	601	585	11	97.33	1.83	140.13
4096 × 942	35	601	585	11	97.33	1.83	140.13
4096 × 1164	10	682	568	37	83.28	5.4	151.28
4096 × 1164	20	682	623	21	91.34	3.07	151.28
4096 × 1164	30	682	654	9	95.89	1.31	151.28
4096 × 1248	10	739	627	41	84.84	5.54	157.06
4096 × 1248	20	739	699	19	94.59	2.57	157.06
4096 × 1248	30	739	701	11	94.86	1.49	157.06

gives the performance of the proposed method for three different image sizes. As shown in Table 3, the highest sensitivity and the lowest FPR are achieved when the number of training samples is 30, which means using 30 defect-free images is the best choice for creating good template images. Moreover, the average sensitivity and average FPR are 96.03% and 1.54%, respectively, while the detection speed is less than 160 ms for each image, which verifies the effectiveness of the proposed method.

4. Conclusions

In this study, a machine vision-based defect detection method was proposed for web offset printing images. To solve the problem of determining the first row, a fast searching technique based on image projection was proposed. The proposed searching method has been used in practical production, and can satisfy the requirement of speed and accuracy for web offset printing. For image registration, a simple yet effective image registration based on the shape context is given. The distinctive characteristic of the shape descriptor is the concave-convex feature of shapes, which can better describe shape contours. For defect detection, the proposed template images can tolerate the differences of pixel values and position differences on contours for defect-free samples. The experimental results demonstrate the effectiveness of the proposed method. To meet the increase in printing machine speed, the speed of detection has room for further improvement in a future study.