A Novel Approach for Foreign Substances Detection in Injection Using Clustering and Frame Difference
Abstract
: This paper focuses on developing a novel technique based on machine vision for detection of foreign substances in injections. Mechanical control yields spin/stop movement of injections which helps to cause relative movement between foreign substances in liquid and an ampoule bottle. Foreign substances are classified into two categories: subsiding-slowly object and subsiding-fast object. A sequence of frames are captured by a camera and used to recognize foreign substances. After image preprocessing like noise reduction and motion detection, two different methods, Moving-object Clustering (MC) and Frame Difference, are proposed to detect the two categories respectively. MC is operated to cluster subsiding-slowly foreign substances, based on the invariant features of those objects. Frame Difference is defined to calculate the difference between two frames due to the change of subsiding-fast objects. 200 ampoule samples filled with injection are tested and the experimental result indicates that the approach can detect the visible foreign substances effectively.1. Introduction
Injection liquids and transfusion solutions have been widely used in clinical treatment in the world. Foreign substances, such as metal shavings, glass shavings and fibres, may appear in injections during the process of production but they are prohibited to appear in qualified injections. These foreign substances can cause serious diseases such as tumor, phlebitis, anaphylactic reaction or even death when they are injected into blood. Despite of such large production and strict standard of injection liquid, most of the pharmaceutical corporations in China still detect the foreign substances artificially [1]: an injection is illumined by a light source, a worker turns it over to checks whether visible foreign substances appear and decides the injection is qualified or not. Though this method is simple, it can result in low accuracy and low efficiency. It is significant to invent an automatic detection method with high accuracy.
Previous works based on machine vision are developed to detect foreign substances. A foreign substances inspection system can be traced back to 1970s [2]. A lot of work is done later. Researchers have developed detection algorithm based on a sequence of frames about the liquid. Ishii et al. [3] breaks up frames into odd ones and even ones, obtains the maximum and minimum values of pixel in odd and even frames respectively, and computes the difference between these values to detect foreign substances. Ishii’s method has a small amount of computation but may be confused by bubbles. Lu et al. [4] proposes another method which use Mean Shift to track the moving object and judge foreign substances by trajectory. Lu’s method can distinguish foreign substances and bubbles effectively but is weak in tracking fast-moving objects. Zhou et al. [5] depicts a classifier (SVM) to judge foreign substances and bubbles based on area size and ratio of length to width. This classifier has a better precision than Lu’s method but it mainly depends on samples. Moghadas et al. [6] uses MLP neural network and SVM to detect foreign substances. Moghadas divides the foreign substances into 4 groups and recognizes surface objects, foreign substances and bubbles efficiently without rotating the bottles. However, this method may miss objects which falls to the bottom before detection. Xiao et al. [7] detects movement by calculating the difference among three contiguous frames, tracks the movement by Particle Filter (PF) and recognizes foreign substances based on the direction of movement. This method does well in slowly-moving objects while it is weak in computation and fast-moving objects.
In this paper, we developed different methods corresponding to different foreign substances. Moving-object Clustering (MC) is proposed to detect subsiding-slowly objects. Clustering [8] is an unsupervised learning method, breaking up a set of observations into clusters in which observations are similar in some sense. In other words, clustering is to make observations in same cluster to be as similar as possible and the ones in different clusters to be as different as possible. Therefore we employ several features to calculate the similarity of moving object and cluster the similar ones of each frame into one cluster. After clustering, we develop some criterions to distinguish foreign substances from clusters.
Features of subsiding-fast objects may change rapidly, which reduces the precision of clustering. In this case, we calculate the frame difference between two frames to recognize foreign substances. We analyze the frame difference values of different samples to support our method.
The paper is organized as follows. Section 2 introduces the framework and architecture of the detection system. Section 3 presents the classification of foreign substances. Section 4 presents algorithm of subsiding-slowly foreign substances detection. Section 5 proposes algorithm of subsiding-fast foreign substances detection. Section 6 is devoted to the experiments and analyses of the results. Section 7 concludes this paper.
2. Framework and Architecture
In order to gain relative movement and reduce the appearance of bubbles, we spin ampoule at high speed and then stop it suddenly. In this case, we can distinguish moving foreign substances from static ampoule bottles. An ampoule filled with injection is fixed on a turntable driven by a motor at speed 2,000 r/min. We use a white LED back-light illumination because white light is insensitive to the color of injection and back-light illumination can cause obvious contrast between transparent and opaque region. A sequence of frames are captured by an industrial camera with resolution of 2,048 × 1,536. The framework of the detection system is shown in Figure 1.
The overall process of the detection system framework is as follow:
High speed rotating. An injection ampoule is delivered to a turntable by conveyor belt and fixed on it. The turntable is driven by a motor located underneath and revolves rapidly together with the ampoule. A vortex is formed so that the foreign substances can move to the center of the ampoule. The ampoule rotates steady to reduce the probability of the appearance of bubbles.
Sudden stoping. After high-speed rotation for a few seconds, the movement is stopped suddenly. Due to inertia, foreign substances continue moving for a while and have relative movement from the ampoule bottle. Thus before the liquid slows down, the white LED is lit on and the camera begins to capture a sequence of frames as inputs of our detection algorithm.
3. Classification of Foreign Substances
Foreign substances used to be classified into black and white ones [3,4,7,9] when both back-light and bottom-light illumination are used together. In our framework, we only use back-light illumination and propose a new classification of foreign substances: subsiding-slowly ones and subsiding-fast ones. This classification is based on the subsiding speed of foreign substances. Subsiding-slowly foreign substances generally include rubber, hair and fiber, and subsiding-fast foreign substances mainly are glasses and metals. Illustrations of foreign substances, fiber and glass, are shown in Figure 2. Due to their different subsiding speed, the features of the two kinds of objects are different.
3.1. Subsiding-Slowly Foreign Substances
These foreign substances often have light weight and subside slowly after the ampoule stops moving. Therefore, we can assume that the average brightness value, area size and shape of the foreign substance are almost identical between contiguous frames. The distribution of brightness values and area sizes of the moving objects and the noise in a sequence of frames are shown in Figure 3, which is a statistical result after motion detection where moving objects and noise are obtained by considering the background image. The brightness values of those objects are within [115, 140], and area sizes are lower than 600. It is clear that brightness values and area sizes of the same foreign substance in a sequence of frames are close. As we assume the subsiding speed to be low, we use the positions of foreign substances between contiguous frames as another feature. Besides, Hu invariant moments [10] of foreign substance qualify as features for they are invariant to scale, rotation and reflection.
3.2. Subsiding-Fast Foreign Substances
In contrast to the subsiding-slowly objects, the features of subsiding-fast objects are that:
The density of subsiding-fast objects is relatively big, so the objects always move close to the bottom area of ampoule where bubbles hardly appear.
The shape of such object varies in frames.
They stop moving rapidly, generally in 3 or 4 frames.
Two contiguous frames with subsiding-fast foreign substance (glass) is shown in Figure 4. It is clear that the area size and shape of glass are different in two contiguous frames, hence clustering is not effective in detecting subsiding-fast foreign substances.
4. Detection of Subsiding-Slowly Foreign Substances
The detection of subsiding-slowly foreign substances includes three phases: motion detection, moving-object clustering and foreign substances recognition.
4.1. Motion Detection
Motion detection is to separate moving objects from frames. Images captured by camera include ampoule area and environment area, so we need to separate ampoule area from the image so as to reduce calculation. Since the height of ampoule and camera is fixed, we can set the vertical area into a fixed range. Compared with the vertical area, the horizontal area of ampoule varies in every detection. We employ Gaussian smooth and Canny operator to obtain the edges of ampoule, project the image to the horizontal direction, and gain the horizontal area by calculating the maximum value in the left half and right half part of the projection.
After separating the ampoule area from images, we obtain a static background by Algorithm 1. A pixel contributes to the static background if the difference between the brightness values of two contiguous frames is less than δ, then we add this pixel value to the sum total. Each pixel value of static background BG(x, y) is the mean of values added to total.
Input: | |
L frames with M × N pixels; | |
Brightness value of pixel (x, y) in frame i, Bi(x, y); | |
Output: | |
Static background with M × N pixels, BG; | |
1: | for coordinate x = 0 to M − 1 do |
2: | for coordinate y = 0 to N − 1 do |
3: | counter = 0; |
4: | total = 0; |
5: | for frame i = 1 to i = L − 1 do |
6: | if |Bi(x, y) − Bi+1(x, y)| < δ then |
7: | counter + +; |
8: | total = total + Bi(x, y); |
9: | end if |
10: | end for |
11: | BG(x, y) = total/counter; |
12: | end for |
13: | end for |
14: | return BG; |
As static background is obtained above, moving objects can be found by computing the difference between each frame and static background, as shown in Equation (1). Motion is described by a binary image MTi and a pixel is considered to be part of a moving object if its value equals to 1.
At last, image morphological transformations is applied to reduce noise and fill up holds.
4.2. Moving-Object Clustering
Generally, based on whether a classification method classifies observations into a finite number of unrelated supervised classes or unsupervised clusters, the classification method is either supervised or unsupervised [11–14]. Clustering, also called exploratory data analysis, is unsupervised classification. The task of clustering is to split up a set of observations into clusters so that the similarity between observations within a cluster is larger than that between observations belonging to different clusters [11]. There are two types of clustering techniques: partitional clustering and hierarchical clustering [15,16].
Hierarchical clustering algorithms produce a nested series of partitions [8], find successive clusters using previously established clusters. Basically, these algorithms are either agglomerative (bottom-up) or divisive (top-down). Agglomerative algorithms separate each element as a cluster and then merge them into larger clusters. Divisive algorithms regard all the elements as a cluster and then divide it into smaller clusters. Figure 5 illustrates the operation of a bottom-up hierarchical clustering using a two-dimensional data set. Figure 5(a) describes 8 data (A–H) in four clusters. A dendrogram yielded by a hierarchical clustering algorithm represents the nested grouping based on the pattern and similarity of data, as shown in Figure 5(b). Different clusters can be obtained by breaking the dendrogram into different leaves [8].
Now come back to our detection, the proposed motion detection in Section 4.1 has marked all the suspected moving objects. After that, a hierarchical clustering algorithm is used to cluster the similarity of moving objects. Foreign substances recognition is then operated on the clusters.
As mention in Section 3.1, several features of subsiding-slowly foreign substances is almost invariant between contiguous frames. Moreover, our clustering algorithm is based on assumptions as follow: (1) The average brightness values, area sizes, Hu moments of one actual moving object in contiguous frames are analogous, and the positions of the object in contiguous frames are close. (2) The similarity of the features of the same actual object in the frames have negative correlation with the distance of the frames containing the object. (3) Suspected objects of one frame do not belong to the same actual object.
Based on the assumptions above and experiments, similarity between suspected objects m and n is defined in Equations (2)–(6):
The clustering algorithm is depicted as follow:
Suppose we have L contiguous frames, each frame i, i ∈ [1, L], has Ni suspected objects, and each object ni in each frame has K features. Therefore, K-dimension eigenvector of object is represented by Oin = {fi,n,1, fi,n,2, …, fi,n,K }, where fi,n,K is the feature k of object n in frame i. All the suspected objects form a matrix in Equation (8):
Using OS as input, we operate bottom-up hierarchical clustering algorithm.
Figure 6 shows the dendrogram of one clustering on a sequence frames, the nested relationship between objects and the average DS in one cluster. There are 292 suspected objects. Although there are still 50 clusters where average DS is 0.85, only a few clusters have objects which appear in each frame. We can rule out most clusters based on this.
4.3. Foreign Substances Recognition
Now we have clusters about noise, bubbles, and foreign substances. Because foreign substances subside slowly, we can assume that these foreign substances and bubbles receive constant force and have uniformly accelerated motion, while the motion of noise is irregular. Furthermore, bubbles always float up to injection surface while foreign substances go down to bottom or travel smaller vertical distance than bubbles. Therefore, we can recognize foreign substances based on motion calculated as follow:
First of all, centroid of objects is gained by Equation (9):
In L frames, we can get (L − 3) third-order derivatives of object positions. Add the square of these derivatives up to be TD in Equation (13).
A threshold η is set to recognize actual objects and noise. It is considered as an actual object if TD < η and as noise if otherwise. Besides, we use the distance on vertical direction to separate foreign substances and bubbles. A threshold μ is set to complete this job.
5. Detection of Subsiding-Fast Foreign Substances
As analyzed in Section 3.2, detection algorithm of subsiding-fast foreign substances is different from that of subsiding-slowly foreign substances. Due to the features of subsiding-fast objects, Frame Difference is proposed. Assume the bottom area of an ampoule in frame i is Gi shown in Equation (14):
After image preprocessing like Gaussian smoothing, we calculate Frame Difference D1 between frame 1 and frame 2, because the different between those frames may be the biggest among all contiguous two frames. At last, a threshold θ is set to recognize subsiding-fast foreign substances. In order to analyze the effect of D, we employ 112 ampoule samples among which 28 samples have glass. Figure 7 shows the statistical result of D. It is obvious that the D value of ampoules with subsiding-fast foreign substances is much bigger than that of ampoules without foreign substances. θ is set to be 150 to detect subsiding-fast foreign substances.
6. Experimental Result
Before testing our system, we have to solve a problem: decide the values of parameters mentioned above. δ in Equation (1) is used to separate foreground from background and decided by experiment. Figure 8 illustrates one of the experimental result. When δ = 1, there will be too much noise in foreground which only increases the computation of detection. When δ = 7, boundary area of foreign substance is lost which affects the features of foreign substances. We choose δ = 3 rather than δ = 5 to increase the precision of motion detection.
α and β are scale parameters used to reduce the contribution of Hu moments on similarity and adjust the pixel distance to the ampoule size in image. They are set to be and .
Our method uses μ in Section 4.3 to recognize foreign substances and bubbles. According to Pan’s simulation [17], the speed of bubbles with over 40 μm radius rises to over 3 mm/s after 0.01 second. The camera captures 9 frames every second and 1 mm distance has 60 pixels. Assume that the speed of bubbles is 3 mm/s, so bubbles can move up at least 20 pixel between two frames. Since some foreign substances hardly move in frames, we set μ to be −10. If one object moves up less than 10 pixel, it is considered to be foreign substance.
η in Section 4.3 is proposed to recognize foreign substances and noise. In order to decide the value of η, we compute TD values in Equation (13) using 603 trace samples as shown in Figure 9. 206 trace samples of foreign substances are used, whose TD values are all lower than 100. Over 99.7% percent of TD values of noise samples are higher than 100. It is obvious that TD can recognize traces of foreign substances and that of noise effectively, therefore we choose η to be 100.
After choosing the value of the parameters, some experiment are made and the results are shown in Tables 2 and 3. There are 200 ampoule samples, among which 129 samples are without foreign substance, 41 samples contain subsiding-slowly foreign substances and the others contain subsiding-fast foreign substances. The accuracy of recognizing subsiding-fast foreign substances is 96.67%. It is because some glass are big and heavy, and they stop moving before frames are captured. In this case, Frame Difference D is lower than θ. Some improvement about control system, motor and camera can solve this problem. Accuracy of subsiding-slowly foreign substances detection is 97.56%. The reasons for failed detection are that:
foreign substances move to the side wall of an ampoule and then miss in a frame;
noise due to camera and environment cause the failure;
foreign substances move too slowly, so they are considered to be background.
The detection precision of ampoules without foreign substance is 98.45%. Since the variation of brightness in some frames is apparent due to the camera and environment, ampoules without foreign substance is recognized into those with subsiding-fast foreign substances.
The comparison among our system and other systems is shown in Table 4. The qualified liquids’ detection accuracy of our system is the highest among the systems brought forward. The accuracy of our system in detecting unqualified liquids is the second best. It is shown that our system can detect qualified and unqualified injections effectively.
We apply Knapp–Kushner testing programs [18], which are accepted by U.S. Food & Drug Administration (FDA) and European pharmacopoeia, to test our automatic detection system. The testing samples are the same as those in Table 2. Because the judging criterion is FQB/FQA > 1, the result shown in Table 5 points out that our detection machine is more effective than workers.
7. Conclusions
This paper mainly studies the detection of foreign substances in injection ampoule. A mechanical system is designed and achieved. We build the system with an industry camera, a motor and some supporting tables like turntable. Foreign substances are classified into two categories: subsiding-slowly ones and subsiding-fast ones. Since we detect foreign substances based on movement, classifying those objects based on their movement features is reasonable. Different characteristics between those categories lead to different detection methods. Moving-object clustering is proved to be effective in detecting subsiding-slowly objects. Four features are employed to calculate the similarity between objects in clusters. Bubbles hardly affect the detection of subsiding-fast objects, so a light computation method Frame Difference is proposed.
Future work may focus on the improvement of mechanical system, algorithms and application of the system in production.
Acknowledgments
This work was partially supported by Grant No. 1118021013 from the Chinese Fundamental Research Funds for the Central Universities, Grant No. BK2010391 from Natural Science Foundation of Jiangsu Province and Grant No. 61100111 from Natural Science Foundation of China.
References
- Du, XP. The difference between artificial inspection and machine inspection. Mech. Elect. Inform 2010, 29, 21–25. [Google Scholar]
- Nagamatsu, K; Tomita, A. Research on the automatic inspection machine of ampoules. J. Takeda Res. Lab 1974, 33, 43–54. [Google Scholar]
- Ishii, A; Mizuta, T; Todo, S. Detection of Foreign Substances Mixed in a Plastic Bottle of Medicinal Solution Using Real-Time Video Image Processing. Proceedings of Fourteenth International Conference on Pattern Recognition, Brisbane, Australia, 16–20 August 1998.
- Lu, J; Wang, YN; Zhang, H; Zhou, BW. On-Line Detection of Foreign Substances in Glass Bottles Filled with Transfusion Solution Through Computer Vision. Proceedings of 2008 International Conference on Information and Automation, Zhangjiajie, China, 20–23 June 2008.
- Zhou, BW; Wang, YN; Ge, J; Zhang, H. A Machine Vision Intelligent Inspector for Injection. Proceedings of Paciia 2008: Pacific-Asia Workshop on Computational Intelligence and Industrial Application, Wuhan, China, 19–20 December 2008.
- Moghadas, S; Rabbani, N. Detection and Classification of Foreign Substances in Medical Vials Using MLP Neural Network and SVM. Proceedings of 6th Iranian Conference on Machine Vision and Image Processing, Isfahan, Iran, 27–28 October 2010.
- Xiao, F; Wang, Y; Zhang, J; Cao, L. Research on an on-line detection system for foreign substances in ampoule. Comput. Measur. Control 2010, 18, 295–298. [Google Scholar]
- Jain, A; Murty, M; Flynn, P. Data clustering: A review. ACM Comput. Surv 1999, 31, 264–323. [Google Scholar]
- Ge, J; Wang, Y; Zhou, B; Zhang, H. Intelligent foreign particle inspection machine for injection liquid examination based on modified pulse-coupled neural networks. Sensors 2009, 9, 3386–3404. [Google Scholar]
- Hu, M. Visual pattern recognition by moment invariants. IEEE Trans. Informa. Theory 1962, 8, 179–187. [Google Scholar]
- Xu, R; Wunsch, D. Survey of clustering algorithms. IEEE Trans. Neural Netw 2005, 16, 645–678. [Google Scholar]
- Kriegel, H; Kr Ger, P; Zimek, A. Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering. ACM Trans. Knowl. Discov. Data 2009, 3, 1–58. [Google Scholar]
- Cherkassky, V; Mulier, F. Learning From Data: Concepts, Theory, and Methods; Wiley-IEEE Press: New York, NY, USA, 2007. [Google Scholar]
- Duda, R; Hart, P; Stork, D. Pattern Classification, 2nd ed; Wiley: New York, NY, USA, 2001. [Google Scholar]
- Berkhin, P. Survey of clustering data mining techniques. In Grouping Multidimensional Data: Recent Advances in Clustering; Springer: London, UK, 2006; pp. 25–71. [Google Scholar]
- Everitt, B; Landau, S; Leese, M. Cluster Analysis; Arnold: London, UK, 2001. [Google Scholar]
- Pan, F; Lu, W. Theoretical study and simulation of a single micro-bubble in still water. Ship Elect. Eng 2009, 29, 155–158. [Google Scholar]
- Knapp, J; Kushner, H. Generalized methodology for evaluation of parenteral inspection procedures. PDA J. Pharm. Sci. Tech 1980, 34, 14–61. [Google Scholar]
object | eigenvector
| ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
B | A | x | y | H[1] | H[2] | H[3] | H[4] | H[5] | H[6] | H[7] | |
O11 | f1,1,1 | f1,1,2 | f1,1,3 | f1,1,4 | f1,1,5 | f1,1,6 | f1,1,7 | f1,1,8 | f1,1,9 | f1,1,10 | f1,1,11 |
Category | Number of samples | True detection | Accuracy |
---|---|---|---|
Subsiding-fast | 30 | 29 | 96.67% |
Subsiding-slowly | 41 | 40 | 97.56% |
No foreign substance | 129 | 127 | 98.45% |
Total number of samples | 200 |
Number of true detection | 196 |
Detection accuracy | 98.00% |
System | Our system (200 samples) | Lu’ system [4] (180 samples) | Zhou’s system [5] (480 samples) | Xiao’s system [7] (3,500 samples) |
---|---|---|---|---|
Detection accuracy of qualified liquids | 98.45% | 96.10% | 97.00% | 98.32% |
Detection accuracy of unqualified liquids | 97.18% | 91.80% | 98.89% | 96.00% |
FQA1 | FQB1 | FQB/FQA | Criterion | Result |
---|---|---|---|---|
407 | 613 | 1.506 | FQB/FQA > 1 | Detection machine is more effective than workers |
1FQi: quality factor of bottle i.FQi = (n/N) × 10, where n is unqualified times of bottle i, N is total testing times of bottle i.FQA: FQ of workers.FQA = FQA[7,10] = Σ FQAi, only FQAi located in [7, 10] are added.FQB: FQ of detection machine.FQB = FQB[7,10] = Σ FQBi, only FQBi located in [7, 10] are added.
© 2011 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
Share and Cite
Lu, G.; Zhou, Y.; Yu, Y.; Du, S. A Novel Approach for Foreign Substances Detection in Injection Using Clustering and Frame Difference. Sensors 2011, 11, 9121-9135. https://doi.org/10.3390/s111009121
Lu G, Zhou Y, Yu Y, Du S. A Novel Approach for Foreign Substances Detection in Injection Using Clustering and Frame Difference. Sensors. 2011; 11(10):9121-9135. https://doi.org/10.3390/s111009121
Chicago/Turabian StyleLu, Guiliang, Yu Zhou, Yao Yu, and Sidan Du. 2011. "A Novel Approach for Foreign Substances Detection in Injection Using Clustering and Frame Difference" Sensors 11, no. 10: 9121-9135. https://doi.org/10.3390/s111009121
APA StyleLu, G., Zhou, Y., Yu, Y., & Du, S. (2011). A Novel Approach for Foreign Substances Detection in Injection Using Clustering and Frame Difference. Sensors, 11(10), 9121-9135. https://doi.org/10.3390/s111009121