Number Determination of Successfully Packaged Dies Per Wafer Based on Machine Vision

Abstract

Packaging the integrated circuit (IC) chip is a necessary step in the manufacturing process of IC products. In general, wafers with the same size and process should have a fixed number of packaged dies. However, many factors decrease the number of the actually packaged dies, such as die scratching, die contamination, and die breakage, which are not considered in the existing die-counting methods. Here we propose a robust method that can automatically determine the number of actual packaged dies by using machine vision techniques. During the inspection, the image is taken from the top of the wafer, in which most dies have been removed and packaged. There are five steps in the proposed method: wafer region detection, wafer position calibration, dies region detection, detection of die sawing lines, and die number counting. The abnormal cases of fractional dies in the wafer boundary and dropped dies during the packaging are considered in the proposed method as well. The experimental results show that the precision and recall rates reach 99.83% and 99.84%, respectively, when determining the numbers of actual packaged dies in the 41 test cases.

1. Introduction

Machine vision has been widely used in various fields because only simple devices are required, and various solutions have been proposed for different kinds of applications, such as industrial measurement [1], text recognition [2], finger recognition [3], medical image analysis [4], face recognition [5], and human computer interfaces [6]. As for the industrial measurement in a semiconductor factory, the automatic optical inspection for wafer-related processes has been widely studied in recent years, such as detecting a die in the open circuit [7], wafer defect detection [8], the LED grading system [9], and rotation calibration in wafer packaging [10].

The number of packaged dies is important information for the manufacturer. In general cases, the number of dies per wafer can be determined by existing counting algorithms [11,12,13,14], which can be found on websites [15] and [16] as well. In these methods, the number of gross dies per wafer can be estimated by entering the parameters such as wafer size, die width, die height, horizontal and vertical spacing. In Reference [11], the variables required in the proposed algorithm include the die dimensions (width and length), the size and orientation of the wafer flat, the size of the non-yielding periphery zone, and the die array relative the center of the wafer. The use and evaluation of the yield models in IC manufacturing are discussed in Reference [12]. Several die-per-wafer yield models, which are derived from Poisson, Binomial, and exponential probability density functions, are reviewed. Moreover, de Vries pointed out that there are different formulas used by different semiconductor manufacturing companies [13]. He found that the accuracy of an exact count algorithm depends on the die area and the aspect ratio. All the above methods require the input of the dimensions of the die and wafer.

Recently, computer vision based methods have received great attention in terms of detecting wafer die defects [17,18]. However, there are few studies that focus on the automatic counting of the die number in the wafer images without providing the size and dimension information for the dies and wafers. In addition to our preliminary work [19], Xu et al. proposed a wafer die-counting algorithm using the geometric characteristics [20] in the wafer images. In this work, RANSAC (Random Sample Consensus) algorithms [21,22] are used to detect the wafer location and the sawing lines. The incomplete dies at the wafer rim strip containing the circle pixels can be easily detected and are then eliminated. Both high counting accuracy and good computational efficiency can be achieved. However, this method does not deal with cases of fractured and randomly dropped dies on the wafer.

In practical situations, some situations that could lead to incorrect counting results should be considered. For example, with the IC testing programs, some dies in a wafer that do not pass the test will not be packaged. The dies not being packaged are left on the wafer and thus a number of dies that different from those successfully packaged are obtained. General commercial equipment [23] could automatically calculate the number of dies in the above case. However, in the cases of the remaining dies that are broken, displaced, or rotated, they cannot be determined straightforwardly. Thus a manual calculation is required if the actual number is desired. It is very time consuming to obtain the precise number of successfully packaged dies through manual wafer inspection. This number should be exactly the same as that determined from the pick and place robot. The clients of the die-packaging factory can only estimate the number, assuming that no abnormal cases occurred during the process. However, this number is usually less than that of actual packaged dies. If the difference between the two numbers is large, the clients will not accept the result and will ask for an explanation. Therefore, in this study we propose a computer-vision-based method that can automatically determine the exact number of actual packaged dies in the residual wafer image. As opposed to existing methods, the wafer and die parameters such as size, width, and length are not required. In addition, the determined die-per-wafer number can be more accurate when abnormal cases occur during the packaging process.

The remaining part of the paper is organized as follows. In Section 2, the proposed methods with the required image processing algorithms are described. The experimental results are provided in Section 3. Finally, the conclusions are drawn in Section 4.

2. Method

Figure 1 shows the systematic block diagram of the proposed system, which consists of five stages in order to determine the number of packaged dies per wafer: (1) wafer region detection: detect the actual wafer region from the input image; (2) wafer position calibration: calibrate the rotation angle of the input wafer image; (3) die region detection: detect the packaged die region from the wafer region; (4) detection of die sawing lines: determine vertical and horizontal segmentation lines in the wafer region, and then use them to partition off the rectangular die regions; (5) die number counting: determine the exact number of actual packaged dies. The detailed procedures are described in later subsections.

Figure 1. The system flowchart of the proposed method.

Figure 1. The system flowchart of the proposed method.

2.1. Wafer Region Detection

Figure 2a shows the input wafer image, which has redundant background information that should be removed for further processing. The detailed steps include: (1) color to grayscale transformation and binarization; (2) region labeling; (3) position calibration.

Figure 2. (a) Original image; (b) grayscale image; (c) histogram of the grayscale image; (d) binarized image.

Figure 2. (a) Original image; (b) grayscale image; (c) histogram of the grayscale image; (d) binarized image.

2.1.1. Color-to-Grayscale Transformation and Binarization

A color image is composed of red, green, and blue components. Each component has a one-byte resolution, i.e., 256 grayscale levels. A color image can be transformed to a grayscale image $i_{\text{gray}}(x,y)$ by using the linear combination method shown in Equation (1).

$$i_{gray}\left(x,y\right)=0.299\times i_r(x,y)+0.587\times i_g(x,y)+0.114\times i_b(x,y)$$

(1)

where i_r, i_g, and i_b denote the grayscale values in red, green, and blue components, respectively, and (x,y) denotes the pixel coordinates.

The binarization process is often used in the object detection and segmentation framework in image processing. The main idea is to select a threshold T_w and to force all the pixel values into two fixed grayscale values, 0 and 255, as shown in Equation (2).

$$w_b\left(x,y\right)=\{\begin{array}{c}0, \mathrm{if} i_{\mathrm{gray}}\left(x,y\right)>T_w\\ 255, \mathrm{Otherwise} \end{array}$$

(2)

where w_b(x,y) is the pixel value after binarization. Figure 2b shows the grayscale image transformed from Figure 2a. Figure 2c shows the histogram of the image in Figure 2b. The threshold T_W can be automatically determined by applying Otsu’s method [24] on the histogram of the grayscale image. Figure 2d shows the binarized result of Figure 2b, based on Equation (2). The white wafer region is well extracted from the black background.

2.1.2. Region Labeling

In order to remove the background and keep the wafer information simultaneously, the mask must conform to the sharpness of the wafer. In Figure 2d, there are certain small regions (black points) in the white region boundary and they should be removed. The connected component labeling method [25] is used to determine the small black regions and the size of each region. Only the largest black area can be retained with the operation shown in Equation (3).

$$w_b^{’}(x,y)=\{\begin{array}{c}0, \mathrm{if} w_{\mathrm{label}}\left(x,y\right)=t\\ 255, \mathrm{Otherwise} \end{array}$$

(3)

where t is the label of the maximum black region and $w_{\text{label}}$ is a matrix that denotes the labeling result. Figure 3a,b show the mask $w_b^{’}$ and the final wafer image $w$ without the background information, respectively. As shown in Equation (4), the final wafer image is obtained as passing the input wafer image onto the mask $w_b^{’}$, i.e.,

$$w\left(x,y\right)=i\left(x,y\right) {{\displaystyle \cap }}^{\text{}} w_b^{’}\left(x,y\right)$$

(4)

Figure 3. (a) The mask of capturing the wafer image; (b) Wafer image with black background.

Figure 3. (a) The mask of capturing the wafer image; (b) Wafer image with black background.

2.2. Wafer Position Calibration

Due to the possible deviation on the wafer placement, the die sawing lines could not be exactly horizontal and vertical in the image, which is necessary in order to accurately determine the die number in the following processing steps. Here, the Canny edge detection [26] and Fourier transform methods are used to determine the wafer rotation angle. First, a Canny edge detector is utilized to obtain the edge information $w_e(x,y)$ in the wafer image shown in Figure 3b. Figure 4a shows that many horizontal and vertical sawing lines appear in this image. This edge information can be transformed into the frequency domain W_e(u,v) by using the two-dimensional Fourier transform shown in Equation (5).

$$W_e\left(u,v\right)={{\displaystyle \iint }}_{-\infty }^{\infty }{w}_e(x,y)e^{-j2\text{π}(ux+vy)}dxdy$$

(5)

Figure 4b shows the corresponding spectrum image, $W_e\left(u,v\right)$, in which the major components appear at the nearly horizontal and vertical directions. In order to determine the angle $\theta$ rotated from the exact horizontal and vertical axes, the threshold Tc shown in Equation (6) is used to binarize the spectrum image $W_e\left(u,v\right)$. Figure 4c shows the binary image $C_t\left(u,v\right)$ derived from the image shown in Figure 4b.

$$C_t\left(u,v\right)=\{\begin{array}{c}0, \mathrm{if} W_e\left(u,v\right)>T_c\\ 255, \mathrm{Otherwise} \end{array}$$

(6)

To determine the rotation angle of the wafer, two points $C_t\left(u_h,v_h\right)$ and $C_t\left(u_l,v_l\right)$ located on the points in the nearly vertical direction in the image $C_t\left(u,v\right)$ are selected. By using Equation (7), the deviation angle $\text{θ}$ can be determined as

$$\theta =\frac{\pi }{2}-ta{n}^{-1}\left(\frac{v_h-v_l}{u_h-u_l}\right)$$

(7)

The deviation angle $\text{θ}$ can be used to calibrate the wafer image $w\left(x,y\right)$ such that the sawing lines are as perfectly horizontal and vertical as possible. Equation (8) shows that the new coordinates (x’, y’) of the rotated image W’(x’, y’) can be determined from the coordinates (x,y). However, if the coordinates (x’, y’) are non-integers, the bilinear interpolation is utilized to obtain the integer coordinates. Figure 5 shows the rotated image after using the position calibration.

$$\left[\begin{array}{c}{x}^{’}\\ y^{’}\end{array}\right]=\left[\begin{array}{cc}cos\text{ θ}& -cos\text{ θ}\\ sin\text{ θ}& sin\text{ θ}\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]$$

(8)

Figure 4. (a) Canny edge detection result of Figure 3b; (b) spectrum of the edge image shown in Figure 4a; (c) binary image in which the black points are corresponding to strong white points shown in Figure 4b.

Figure 4. (a) Canny edge detection result of Figure 3b; (b) spectrum of the edge image shown in Figure 4a; (c) binary image in which the black points are corresponding to strong white points shown in Figure 4b.

Figure 5. The wafer-region image after the position calibration.

Figure 5. The wafer-region image after the position calibration.

2.3. Die region Detection

After the wafer region has been determined and its position has been calibrated in Subsection 2.2, the resident die regions, which are not taken away during packaging, are detected. In order to avoid the color saturation problem caused by the reflective points in the RGB color space, the YCbCr color space is utilized such that the brightness can be separated from its color components. Equation (9) shows the transformation function.

$$\left[\begin{array}{c}Y\\ Cb\\ Cr\end{array}\right]=\left[\begin{array}{c}16\\ 128\\ 128\end{array}\right]+\left[\begin{array}{ccc}65.481& 128.553& 24.966\\ -37.797& -74.203& 112\\ 112& -93.786& -18.214\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right]$$

(9)

Figure 6a–c show the Y, Cb, and Cr components of the wafer image in Figure 5, respectively. As shown in Figure 6b, the Cb component has a more obvious contrast than the other components and is nearly unaffected by reflection. To extract the regions of the resident dies, a binarization process defined in Equation (10) is used.

$$W_{\mathrm{dies}}\left(x,y\right)=\{\begin{array}{c}255, \mathrm{if} Cb\left(x,y\right)<T_{cb} \\ 0, \mathrm{Otherwise} \end{array}$$

(10)

where Cb(x,y) denotes the pixel value in the Cb component, T_cb is the threshold value, which is determined by applying Otsu’s method on the histogram of the Cb component image shown in Figure 7a; and $W_{\text{dies}}\left(x,y\right)$ denotes the resident die image shown in Figure 7b.

Figure 6. (a) The Y component image; (b) the Cb component image; (c) the Cr component image in the YCbCr color space.

Figure 6. (a) The Y component image; (b) the Cb component image; (c) the Cr component image in the YCbCr color space.

Figure 7. (a) Histogram of the Cb component; (b) the image of the resident die regions.

Figure 7. (a) Histogram of the Cb component; (b) the image of the resident die regions.

As shown in Figure 5, the rim strip of the wafer contains no dies and should be excluded to avoid possible confusion on die counting in further processing. In the third step, the area of the wafer region which does not include the rim strip is determined. The four non-zero value points (x_a,y_a), (x_b,y_b), (x_c,y_c), and (x_d,y_d) corresponding to the top, bottom, leftmost, and rightmost boundary points, respectively, are selected in the image $W_{\text{dies}}$ shown in Figure 7b. The four points can be used to define an ellipse mask region $W_D$ using Equation (11).

$$W_D\left(x,y\right)=\{\begin{array}{c}255, \mathrm{if} \frac{{\left(x-h\right)}^2}{{\text{α}}^2}+\frac{{\left(y-k\right)}^2}{{\text{β}}^2}\le 1\\ 0, \mathrm{Otherwise} \end{array}$$

(11)

where

$$h=\frac{x_a+x_b}{2}, k=\frac{y_a+y_b}{2}, \alpha =\frac{x_d-x_c}{2}, \beta =\frac{y_a-y_b}{2}$$

Figure 8a shows a typical mask region, which is very close to a circle. By applying this mask onto the image in Figure 5, Figure 8b shows that the mentioned rim strip (shown in the white region) can be successfully removed.

Figure 8. (a) The mask of the wafer image; (b) the wafer image in which the region outside the white contour has been removed.

Figure 8. (a) The mask of the wafer image; (b) the wafer image in which the region outside the white contour has been removed.

2.4. Detection of Die Sawing Lines

Prior to die counting, the geometrical information of a standard die, whose shape is rectangular and is defined by the sawing lines, should be derived. Therefore, detecting the die sawing lines in the wafer image is required. However, the sawing lines shown in Figure 9a are not obvious and thus are difficult to be easily detected. To overcome this problem, the histogram equalization method that can increase the global contrast in the image [27] is used. Figure 9b shows the enhanced result of Figure 9a. In this enhanced image $e\left(x,y\right)$, the sawing lines have been intensified, thus they can be more easily detected. Equations (12) and (13) show that the pixel values of each column and row in the wafer image (Figure 9b) are accumulated to obtain the two vectors $v(j)$ and $h(i)$, respectively.

$$v\left(j\right)={{\displaystyle \sum }}_{x}e\left(x,j\right)$$

(12)

$$h(i)={{\displaystyle \sum }}_{y}e\left(i,y\right)$$

(13)

Figure 9. (a) The grayscale image obtained from Figure 3b; (b) the enhanced image e(x, y) obtained by applying the histogram equalization on Figure 9a.

Figure 9. (a) The grayscale image obtained from Figure 3b; (b) the enhanced image e(x, y) obtained by applying the histogram equalization on Figure 9a.

Figure 10a,b show the projections of the accumulated pixel values in the columns and rows, respectively. In both projections, many obvious peaks appear in accordance with the sawing line positions in the wafer image shown in Figure 9b, because the pixel values in the sawing lines are much larger than that in the die regions. In addition, the intervals between two consecutive peaks should be very similar in these two projections. However, due to the incomplete dies and the non-uniform illumination condition, the peak and non-peak values vary a lot. Therefore, the peaks cannot be detected by directly applying a fixed thresholding scheme. Figure 11 shows the flowchart of the proposed method to detect the peak positions v_p(j) in the vertical direction. Here j denotes the index of the detected peaks. The peak positions h_p(i) in the horizontal direction can be detected with a similar strategy.

Figure 10. The projections of the accumulated values in (a) the columns and (b) the rows.

Figure 10. The projections of the accumulated values in (a) the columns and (b) the rows.

Figure 11. Peak detection flowchart in the vertical direction.

Figure 11. Peak detection flowchart in the vertical direction.

Figure 12 shows the detected die sawing lines in both the vertical and horizontal directions. The peak detection method described above shows that many incorrect sawing lines are detected. To solve this problem, the average line method is proposed to filter out the miss-detected sawing lines. Figure 13a,b show the depicted white lines representing the average values calculated by using Equations (14) and (15), respectively.

$$v’(j)=\frac{{{\displaystyle \sum }}_{j-l}^{j+l}v(j)+v(j)}{l+2}$$

(14)

$$h’(i)=\frac{{{\displaystyle \sum }}_{i-z}^{i+z}h(i)+h(i)}{z+2}$$

(15)

where $v’$ and $h’$ are the average values in the sliding windows of widths. Assume that $l$ and $w$ are the length and width of the die when the threshold T shown in Figure 11 is increased to a larger number in the peak detection method, respectively. As shown in Figure 11, a larger threshold T indicates a higher difference values between the two consecutive bins in the histogram. Figure 14a shows the detected peak positions with a large T value. The values l and z are calculated by using Equations (16)–(19), where we define the shortest distant between two peak positions to be length and width of the die in the histogram with a T value.

$$l=min\left(D_h\left(i\right)\right)$$

(16)

$$z=min\left(D_v\left(j\right)\right)$$

(17)

$$D_v\left(j\right)=\widehat{v_p}\left(j+1\right)-\widehat{v_p}\left(j\right)$$

(18)

$$D_h\left(i\right)=\widehat{h_p}\left(i+1\right)-\widehat{h_p}\left(i\right)$$

(19)

where $\widehat{v_p}$ and $\widehat{h_p}$ denote the detected peaks positions. $D_v$ and $D_h$ denote the distances between two consecutive peak positions in the vertical and horizontal directions, respectively. The die sawing lines v_p and h_p detected in the peak detection method are compared to the average values $v’$ and $h’$, respectively. If the condition v_p (j) > $v^{’}(j)$ is satisfied, the new die sawing line is recorded as v_r(k) = v_p(j). Figure 14b shows the detection result based on this average line method.

Figure 12. Result of the die sawing line detection.

Figure 12. Result of the die sawing line detection.

Figure 13. The result of the average line method: (a) average values shown on a white line in the vertical direction; (b) average values shown on a white line in the horizontal direction.

Figure 13. The result of the average line method: (a) average values shown on a white line in the vertical direction; (b) average values shown on a white line in the horizontal direction.

The last step is to guarantee that the distance between the adjacent real wafer saw lines v_r(k) and v_r(k+1) is greater than a predefined value z. Figure 15 shows the flowchart of this process and Figure 16 shows the final result of the die sawing lines.

2.5. Counting the Packaged Dies Number

With the detected die sawing lines shown in Figure 16, the exact number of successfully packaged dies can be determined. In a normal case, the dies that have been taken away from the wafer can be determined. In special cases, some dies are broken, rotated, or displaced during the wafer sawing, wafer probing, or packaging processes. In counting the number of regular rectangular dies, both the normal and special cases must be considered.

Figure 14. (a) The result of the peak detection method with a T value; (b) the result of the average line method.

Figure 14. (a) The result of the peak detection method with a T value; (b) the result of the average line method.

2.5.1. Normal Cases

Figure 17 shows that a matrix is constructed based on the die sawing lines. The binary entry value of the matrix is determined using Equation (20). The main idea is to evaluate the size of the wafer image $W_D$ in each corresponding local area, $N_c(m,n)$ and to use the threshold $T_{n1}$ for judging whether the die in this location has been packaged or not.

$$N_c\left(m,n\right)=\{\begin{array}{c}1, \mathrm{if} {\displaystyle \sum _{x=v_f\left(m\right)}^{v_f\left(m+1\right)}}{\displaystyle \sum _{y=h_f\left(n\right)}^{h_f\left(n+1\right)}}\frac{W_D\left(x,y\right)}{255}>T_{n1} \\ 0, \mathrm{Otherwise}\end{array}$$

(20)

where the coordinate (x,y) denotes the pixel position in the image, the coordinate (m,n) denotes the entry position in the matrix, and N_c denotes the area corresponding to the size of dies shown in Figure 18a. Some special dies, which should be taken away but in fact are not, or the opposite case, are also considered here. The information of the resident die image W_dies is utilized to identify the actual die number N in the normal case. Equations (21) and (22) show the steps to determine the number N. Figure 19b shows the resultant image in which the incomplete dies have been discarded.

$$\widehat{N_c}\left(m,n\right)=\{\begin{array}{c}1, \mathrm{if} N_c\left(m,n\right)=1 \text{and} {\displaystyle \sum _{x=v_f\left(m\right)}^{v_f\left(m+1\right)}}{\displaystyle \sum _{y=h_f\left(n\right)}^{h_f\left(n+1\right)}}\frac{d\left(x,y\right)}{255}<T_{n2}\\ 0, \mathrm{Otherwise}\end{array}$$

(21)

$$N={{\displaystyle \sum }}_m{{\displaystyle \sum }}_n\widehat{N_c}\left(m,n\right)$$

(22)

where $T_{n2}$ is the threshold used to judge the location of the die which has been packaged or not, N denotes the number of the packaged dies in the normal case, and $d\left(x,y\right)$ denotes the calibrated wafer image shown in Figure 5.

Figure 15. The flowchart of the last step.

Figure 15. The flowchart of the last step.

Figure 16. The final result of the detected die sawing lines.

Figure 16. The final result of the detected die sawing lines.

Figure 17. Wafer image mapped in the matrix defined by using the detected die sawing lines.

Figure 17. Wafer image mapped in the matrix defined by using the detected die sawing lines.

Figure 18. (a) The area that is judged to be packaged; (b) actual packaged area in the normal case.

Figure 18. (a) The area that is judged to be packaged; (b) actual packaged area in the normal case.

2.5.2. Abnormal Cases

If the resident dies are broken, rotated, or shifted from the normal position, they are abnormal cases, and should be treated in another way. Consider the calibrated wafer image shown in Figure 5. First, the intersection of the calibrated image and the actual packaged die image shown in Figure 18b is determined. The dies belonging to the abnormal case can be determined using Equation (23).

$$S_a\left(x,y\right)=d\left(x,y\right){{\displaystyle \cap }}^{\text{}}\widehat{N_c}\left(x,y\right)$$

(23)

where $S_a$ denotes the image of the abnormal case shown in Figure 19a. It contains some deviation caused by the error introduced during the die sawing lines detection. Because the size of the deviation area is smaller than that of a normal die, the deviation could be removed by using the steps described in Equations (24) and (25).

$$S_c\left(m,n\right)=\{\begin{array}{c}1, \mathrm{if}{\displaystyle \sum _{i=v_f\left(m\right)}^{v_f\left(m+1\right)-1}}{\displaystyle \sum _{j=h_f\left(n\right)}^{h_f\left(n+1\right)-1}}{S}_a\left(i,j\right)/255 <T_{s1} \mathrm{and} S_n\left(m,n\right)=1\\ 0, \mathrm{Otherwise}\end{array}$$

(24)

$$S_n\left(m,n\right)=\{\begin{array}{c}1, \mathrm{if} \widehat{N_c}\left(m-1,n\right)<T_{n1} or \widehat{N_c}\left(m+1,n\right)<T_{n1} \\ \mathrm{or} \widehat{N_c}\left(m,n-1\right)<T_{n1} or \widehat{N_c}\left(m,n+1\right)<T_{n1}\\ 0, \mathrm{Otherwise}\end{array}$$

(25)

here $S_c$ denotes the result after removing the small deviation regions. The condition $S_n\left(m,n\right)=1$ indicates the position that contained the deviation value. Figure 19b shows the resultant image $S_n$.

Figure 19. (a) The dies with small deviation regions in the abnormal case; (b) the result after removing the small deviation regions; (c) the final dies in the abnormal case.

Figure 19. (a) The dies with small deviation regions in the abnormal case; (b) the result after removing the small deviation regions; (c) the final dies in the abnormal case.

In the second step, the small areas are filtered and the number S of dies is calculated by utilizing the labeling method. The result is shown in Figure 19c. Finally, the number of the successfully packaged dies is obtained by using Equation (26).

$$P=N-S$$

(26)

3. Experimental Results

In our experiments, a Microsoft LifeCam Cinema image sensor was used, and the computer specifications were as follows: CPU: Intel Core i5-3570K, RAM: DDR3 1600MHz 4G. The programming was implemented using Visual Studio 2008. Figure 20 shows the photographic environment. Two illumination sources were used in the experimental set up to reduce reflection effects in the captured images. According to the empirical tests in our experiments, the various threshold values in the proposed method are given as follows: both T_w and T_cb are automatically determined by using the Otsu method, T_c = 185, T = 25, T_n1 = 800, T_n2 = 700, and T_s1 = 450. The accuracy for evaluating the proposed method is defined in two aspects, the recall and precision rates, which are expressed as

$$\text{Precision rate}=\frac{\text{Number of correctly deteced dies}}{\text{Total number of detected dies}}$$

(27)

$$\text{Recall rate}=\frac{\text{Number of correctly detected dies}}{\text{Total number of dies}}$$

(28)

Figure 20. The experimental setup for taking the wafer images.

Figure 20. The experimental setup for taking the wafer images.

Figure 21a–j show ten samples of the all test wafer images in our experiments. Both normal and abnormal cases of the resident dies can be found in these images. Figure 22a–j show the corresponding detection results of the die regions. Figure 23a–j show the results of the detected die areas in the normal cases, while Figure 24a–j show the results of the detected dies area in the abnormal cases. Some tiny parts of the die, as shown in Figure 22b,c,f,g, can be discarded to prevent the inclusion of the number in the abnormal cases.

Figure 21. The ten test wafer images numbered from (a) to (j).

Figure 21. The ten test wafer images numbered from (a) to (j).

Figure 22. The results of the die region detection corresponding to the images shown in Figure 21a–j.

Figure 22. The results of the die region detection corresponding to the images shown in Figure 21a–j.

Figure 23. The packaged regions in the normal cases corresponding to the images shown in Figure 22a–j.

Figure 23. The packaged regions in the normal cases corresponding to the images shown in Figure 22a–j.

Figure 24. The results of the die regions in the abnormal cases corresponding to the images shown in Figure 22a–j.

Figure 24. The results of the die regions in the abnormal cases corresponding to the images shown in Figure 22a–j.

There are 41 wafers images in total. The die numbers are divided into three situations: normal, abnormal, and packaged. As shown in this table, most of the dies in the abnormal cases can be correctly detected. Table 1 summarizes the determined numbers in the normal, abnormal, and actual packaged cases. The recall and precision rates are provided as well. In this table, the precision and the recall rates are (P/G) × 100% and 100%, respectively, when the detected number P is greater than the ground truth number G. On the contrary, the precision and the recall rates are 100% and (P/G) × 100%, respectively, when the detected number P is less than the ground truth number G. If the two numbers are equal (P = G), then both the precision and the recall rates are 100%. The recall and precision rates in the experimental results of the 41 test images achieve 99.82% and 99.84%, respectively. The computation time for each test image is around 0.7 s in average.

Table 1. The recall and precision results in the experiments.

Wafer Image	Normal (N)	Abnormal (S)	Packaged (P)	Ground Truth (G)	Precision Rate	Recall Rate
1	315	3	312	311	99.68%	100.00%
2	315	3	312	311	99.68%	100.00%
3	315	5	310	309	99.68%	100.00%
4	310	10	310	309	99.68%	100.00%
5	323	13	310	309	99.68%	100.00%
6	314	6	308	309	100.00%	99.68%
7	313	3	310	310	100.00%	100.00%
8	314	4	310	310	100.00%	100.00%
9	318	4	314	314	100.00%	100.00%
10	321	7	314	314	100.00%	100.00%
11	326	11	315	314	99.68%	100.00%
12	326	11	315	315	100.00%	100.00%
13	312	1	311	311	100.00%	100.00%
14	323	10	313	311	99.36%	100.00%
15	324	12	312	311	99.68%	100.00%
16	315	3	312	311	99.68%	100.00%
17	312	1	311	311	100.00%	100.00%
18	313	1	312	311	99.68%	100.00%
19	319	7	312	311	99.68%	100.00%
20	321	6	315	316	100.00%	99.68%
21	320	5	315	316	100.00%	99.68%
22	320	2	318	318	100.00%	100.00%
23	321	4	317	318	100.00%	99.69%
24	312	1	311	311	100.00%	100.00%
25	319	7	312	311	99.68%	100.00%
26	313	1	312	311	99.68%	100.00%
27	317	8	309	311	100.00%	99.36%
28	317	7	310	311	100.00%	99.68%
29	302	4	298	311	100.00%	95.82%
30	317	6	311	312	100.00%	99.68%
31	313	0	313	313	100.00%	100.00%
32	315	2	313	313	100.00%	100.00%
33	315	2	313	313	100.00%	100.00%
34	316	3	314	313	99.68%	100.00%
35	322	7	315	313	99.37%	100.00%
36	318	2	316	316	100.00%	100.00%
37	325	9	316	316	100.00%	100.00%
38	330	11	319	316	99.06%	100.00%
39	320	3	317	316	99.68%	100.00%
40	319	2	317	315	99.37%	100.00%
41	316	1	315	315	100.00%	100.00%

Table 1. The recall and precision results in the experiments.

Wafer Image	Normal (N)	Abnormal (S)	Packaged (P)	Ground Truth (G)	Precision Rate	Recall Rate
1	315	3	312	311	99.68%	100.00%
2	315	3	312	311	99.68%	100.00%
3	315	5	310	309	99.68%	100.00%
4	310	10	310	309	99.68%	100.00%
5	323	13	310	309	99.68%	100.00%
6	314	6	308	309	100.00%	99.68%
7	313	3	310	310	100.00%	100.00%
8	314	4	310	310	100.00%	100.00%
9	318	4	314	314	100.00%	100.00%
10	321	7	314	314	100.00%	100.00%
11	326	11	315	314	99.68%	100.00%
12	326	11	315	315	100.00%	100.00%
13	312	1	311	311	100.00%	100.00%
14	323	10	313	311	99.36%	100.00%
15	324	12	312	311	99.68%	100.00%
16	315	3	312	311	99.68%	100.00%
17	312	1	311	311	100.00%	100.00%
18	313	1	312	311	99.68%	100.00%
19	319	7	312	311	99.68%	100.00%
20	321	6	315	316	100.00%	99.68%
21	320	5	315	316	100.00%	99.68%
22	320	2	318	318	100.00%	100.00%
23	321	4	317	318	100.00%	99.69%
24	312	1	311	311	100.00%	100.00%
25	319	7	312	311	99.68%	100.00%
26	313	1	312	311	99.68%	100.00%
27	317	8	309	311	100.00%	99.36%
28	317	7	310	311	100.00%	99.68%
29	302	4	298	311	100.00%	95.82%
30	317	6	311	312	100.00%	99.68%
31	313	0	313	313	100.00%	100.00%
32	315	2	313	313	100.00%	100.00%
33	315	2	313	313	100.00%	100.00%
34	316	3	314	313	99.68%	100.00%
35	322	7	315	313	99.37%	100.00%
36	318	2	316	316	100.00%	100.00%
37	325	9	316	316	100.00%	100.00%
38	330	11	319	316	99.06%	100.00%
39	320	3	317	316	99.68%	100.00%
40	319	2	317	315	99.37%	100.00%
41	316	1	315	315	100.00%	100.00%

4. Conclusions

This paper presents a machine-vision-based method to automatically determine the number of successfully packaged dies in a wafer image. The most significant contribution of the proposed method is that it can handle all the cases of the residual dies, including those in the wafer boundary, broken dies, fractured dies and dropped dies in the wafer. The precision and recall rates achieved were 99.82% and 99.84%, respectively, in the 41 test wafer images.

Figure 21a–j show that the brightness and position of each wafer image are slightly different. It is influenced by the ambient light and non-fixed photographic environment. In the future work, we will improve the image capture environment such that the number of system parameters can be reduced to make the system simpler. In addition, more test images will be collected to make the proposed method more robust in the various abnormal cases.