A Novel Electronic Chip Detection Method Using Deep Neural Networks

Abstract

Electronic chip detection is widely used in electronic industries. However, most existing detection methods cannot handle chip images with multiple classes of chips or complex backgrounds, which are common in real applications. To address these problems, a novel chip detection method that combines attentional feature fusion (AFF) and cosine nonlocal attention (CNLA), is proposed, and it consists of three parts: a feature extraction module, a region proposal module, and a detection module. The feature extraction module combines an AFF-embedded CNLA module and a pyramid feature module to extract features from chip images. The detection module enhances feature maps with a region intermediate feature map by spatial attentional block, fuses multiple feature maps with a multiscale region of the fusion block of interest, and classifies and regresses objects in images with two branches of fully connected layers. Experimental results on a medium-scale dataset comprising 367 images show that our proposed method achieved $mA{P}^{0.5}=0.98745$ and outperformed the benchmark method.

1. Introduction

Electronic chip assembly is a key link of electronic manufacturing, and its task is to place and solder chips onto printed circuit boards (PCBs). After electronic chip assembly, the chip and PCB are combined for electronic production. In this process, placement error is the distance between the real and ideal positions that causes functional defects in electronic products. With chip detection methods, electronic products with large placement errors can be found as early as possible. Machine vision techniques can also be used to detect chip position without damaging electronic products. Therefore, chip detection methods based on machine vision play an important role in electronic industries.

Our electronic chip detection method was designed to estimate the class and location of chips in a PCB. This is implemented by a electronic chip detection system that integrates various electronic parts. As shown in Figure 1, the detection system consists of four parts: PCB conveyer, image capture module (including camera, lens, and lighter), $x--y$ moving module (not shown in Figure 1 for simplicity), and an industrial PC. The PCB to be detected is first transferred to the center of the chip detection system by the conveyor; the image capture module is moved to several predefined positions and takes pictures of the PCB; lastly, these pictures are processed by an industrial PC to classify and locate chips. Three examples of PCB images are illustrated in Figure 2, and their characteristics are given as follows:

(1)

there are multiple chips in one picture;

(2)

the background of PCB image is complex, including pins, pads, flame retardant layer, and silk screen;

(3)

the size, color, and other characteristics of chips vary greatly.

2. Related Work

To classify and locate chips from images, there are many research achievements on electronic chip detections methods based on machine vision. Crispin [1] incorporated a normalized cross-correlation (NCC) template-matching approach that could reduce computational cost by constraining the search space, and optimize the search strategy of template positions using a genetic algorithm. A tree step algorithm for light-emitting diode (LED) chip localization was proposed by Zhong [2]. First, the positions of potential chips were extracted by applying a image segmentation and blob analyzation method; then, orientations of potential chips were predicted on the basis of dominant orientations; lastly, chips were precisely located using gradient orientation features according to the predicted positions and orientations. Gao [3] proposed a novel algorithm to inspect ball grid array (BGA) component defects: first, a grayscale image of solder balls was extracted with an adaptive thresholding algorithm with modified $(ϵ,\delta )$-component segmentation; then, the ball array was generated with a line-based-clustering method; lastly, the precise position and orientation of BGA were estimated from the recognition results. The main cause of errors in chip detection was analyzed by applying a two-step calibration algorithm in Wang [4]. Zhong [5] proposed a three-step algorithm to exclude polycrystalline and fragmentary LED chips. First, blobs were obtained from an image with a simple but efficient image segmentation algorithm; second, abnormal blobs were excluded, and the position and orientation of a potential object were predicted on the basis of the pose of the minimal enclosing rectangle of each candidate blob; lastly, precise LED chips in the originally captured image were located on the basis of gradient orientation features. Bai [6] studied an online component positioning problem based on corner points that incorporated preprocessing, coarse positioning, and fine positioning stages. The preprocessing stage applied Harris corners and subpixel corner points that were extracted from images of real components. The coarse positioning step used distance and shape feature matching methods to compute correct correspondences between key points and Harris corner points. Lastly, the coarse and fine positioning problems were formulated as least-squares error problems.

With the development of deep-learning theory, object detection methods have recently achieved great success [7,8,9]. Object detection methods based on deep learning can generally be divided into two categories: one- and two-stage methods. One-stage object detection methods directly embed location and classification subnetworks into the font of a main backbone network. Typical one-stage object detection methods include YOLO [10,11,12], SSD [13], and DSSD [14]. Two-stage object detection methods introduce region proposal networks to predict candidate bounding boxes, and estimate class and location with multilayer fully connected networks from these bounding boxes. Typical two-stage object detection methods include R-CNN [15,16,17] and ThunderNet [18]. One-stage object detection methods are faster than two-stage object detection methods, and two-stage object detection methods are more precise than one-stage object detection methods.

Deep-learning-based detection methods come from training a multilayer convolutional neural network from training data; thus, neural network architectures learning features from training data is a research hotspot. Lin [19] designed a two-pathway architecture that contained a top–down and a down–top pathway to extract multiscale hierarchical feature maps. Qin [18] proposed a lightweight architecture to realize real-time object detection. To enrich feature representation, several blocks were introduced in that network, such as the context enhancement module (CEM) and spatial attention module (SAM). Liu [20] proposed a context embedding object detection network to detect concealed objects from millimeter wave images. In context embedding object detect networks, backbone features are attached to tree parallel branches with dilation sizes of 3, 6 and 12 to form the context embedding module and to incorporate surrounding information. Fang [21] fused the semantic object feature extraction module (Conv2dNet), the spatiotemporal feature extraction module (Conv3DNet) and the saliency feature-sharing module to generate the final saliency map for real-time video processing. Wang [22] combined dual-branch feature extraction and gradually refined the cross-fusion module in the network for camouflaged object detection. Gu [23] assembled an X-ray proposal network that applies data augmentation to enlarge input image datasets, and an X-ray discriminative network that fuses region of interest (ROI) feature maps from several levels for baggage inspection. A bidirectional attention feature pyramid network with cosine similarity was proposed for photovoltaic cell defect detection [24].

Table 1. Advantages of existing methods.

Method	Reference	Advantages
Computer-vision-based methods	FTM [2]	Fast template-matching method applied to LED chip localization.
	LBC [3]	Line-based clustering approach applied to BGA component localization.
	VF [4]	Main cause of errors in chip detection was analyzed.
	BATM [5]	Blob-analysis-based template matching method introduced into LED chip detection.
	CPCF [6]	Corner-point-based coarse fine method introduced into chip localization.
Deep-learning-based method	FPN	Feature-pyramid-based feature extraction introduced into object detection.
	Thunder Net [18]	Context-enhancement and spatial-attention modules introduced into object detection.
	COB [20]	Context-embedding module introduced into concealed object detection form millimeter wave image.
	SOD [21]	Semantic object feature extraction module (Conv2dNet), spatiotemporal feature extraction module (Conv3DNet), and saliency feature-sharing module fused for real-time video object detection.
	D2C-Net [22]	Dual-branch feature extraction and gradually refined cross-fusion module fused for camouflaged object detection.
	XRBI [23]	X-ray proposal and X-ray discriminative networks assembled for baggage inspection.
	PCDD [24]	Bidirectional attention feature pyramid network introduced for photovoltaic-cell defect detection.

shows the advantages of existing methods, which have two disadvantages: (1) they cannot detect multiple chips at the same time, and (2) cannot process chip images with a complex background. These drawbacks render them unsuitable for real applications. To solve these problems, we propose a novel chip detection method motivated by [18,23,25].

3. Proposed Methodology

In the electronic industry, the main aim of the proposed electronic chip detection method is to classify and locate chips in images. To overcome these challenges, a novel chip detection method is proposed in this work. Its methodology is composed of three steps: (1) AFF-embedded CNLA and pyramid-feature modules are combined to extract multiscale pyramid feature maps from chip images; (2) candidate bounding boxes are proposed in the region proposal module (RPM); (3) region intermediate feature maps are fused into enhanced feature maps from the spatial attentional block, and chip class and location are estimated by two branches of fully connected layers. The overall structure of the novel chip detection method is demonstrated in Figure 3.

3.1. Feature Extraction Module

In traditional deep-learning-based object detection methods, a multilayer framework that contains a series of convolutional layers is utilized to extract high-level features from images. In the feature extraction framework, each layer takes the output of the lower layer as input and output features to the higher layer as input. The input of the lowest layer is the raw image, and the output of highest layer is used as the final feature of the detection module. To drop memory usage, convolutional layers in the feature extraction framework apply stride to reduce the feature map. This multilayer structure is able to learn feature extraction methods from large-scale training datasets, and its performance exceeds that of handcrafted feature extraction methods. Several research works [23,26] revealed that multilayer feature extraction cannot extract semantic and location information at the same time: semantic information exists in the upper layers but not in the lower layers, and the opposite for location information. As demonstrated in Figure 3, an improved feature pyramid framework was applied to extract image features in our work. Similar to feature pyramid networks (FPNs) [19], the feature extraction module (FEM) consists of two pathways: the bottom–up and up–bottom pathways.

In the proposed improved feature pyramid framework, the bottom–up pathway was designed to extract hierarchical features; hence, traditional multiple convolutional layer structure ResNet was employed, which is composed of 5 stages, and every layer of the same stage had the same output size. To save memory, the output of the first stage was ignored, and the output of remaining stages $\{C_2,C_3,C_4,C_5\}$ was chosen to form the reference set.

Both semantic and location information is essential for object detection, but it is distributed in different layers in the bottom–up pathway. To combine this information, it is necessary to fuse features from different layers in the up–bottom pathway. In the highest layer of up–bottom pathway $p_5$, the highest layer of bottom–up pathway $c_5$ was attached to a $1\times 1$ convolutional layer. In other layers of up–bottom pathway $\{p_i,i=1,...,4\}$, a building block was applied. The building block is illustrated in Figure 4: the feature from the same layer of down–up pathway $c_i$ was attached to a $1\times 1$ convolutional layer, the feature from the higher layer of up–down pathway $p_{i+1}$ was attached to a $2\times$ up layer, and these two features were then fused into feature $p_i$ with the AFF-embedded CNLA block.

AFF-embedded CNLA block in Figure 4 demonstrated as in Figure 5, consisting of two parts: (top) AFF block; (bottom) CNLA block.

The AFF block [25] was designed to combine two input features ${\mathcal{F}}_1$ and ${\mathcal{F}}_2$. To improve detection performance, global feature context (GFC) and local channel context (LCC) were both taken into account in the AFF block.

Considering feature $\mathcal{F}\in {\mathbb{R}}^{C\times H\times W}$ of C channels whose width and height were W and H, the GFC is defined as follows:

$$\mathbf{GFC}\left(\mathcal{F}\right)=\mathbf{BN}({\mathbf{W}}_2·\mathbf{ReLU}\left(\mathbf{BN}({\mathbf{W}}_1·gap\left(\mathcal{F}\right))\right)),$$

(1)

where ${\mathbf{W}}_1$ and ${\mathbf{W}}_2$ are two learnable parameters. $\mathbf{BN}$ in Equation (1) means batch normalization (BN) [27] that is proposed to address the internal covariate shift phenomenon in deep-learning networks. The internal covariate shift phenomenon is caused by the change in the input of each layer, increasing training epochs. In the BN layer, two parameters are introduced to scale and shift normalized values; then, normalization transformation can represent the identified transform. For input $\left\{x_{1...m}\right\}$ over a minibatch, the output of BN layer $y_i=\mathbf{BN}\left(x_i\right)$ is defined as follows:

$$\begin{array}{cc}\hfill {\mu }_B& =\frac{1}{m}\sum _{i=1}^m{x}_i,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\sigma }_B^2& =\frac{1}{m}\sum _{i=1}^m{(x_i-{\mu }_B)}^2,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\widehat{x}}_i& =\frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill y_i& =\mathbf{BN}\left(x_i\right)=\gamma {\widehat{x}}_i+\beta ,\hfill \end{array}$$

(5)

where $\gamma$ and $\beta$ are two learnable parameters, and $ϵ$ is the smallest positive value. $\mathbf{ReLU}$ in Equation (1) denotes rectified linear unit (ReLU) [28], and is an activation function for convolutional networks. The activation function can activate neural layers when the output reaches a predefined threshold, so it transforms input into the required output. ReLU is a nonlinear function that directly outputs the input value if it is positive; otherwise, it outputs zero. Mathematically, the ReLU is defined as follows:

$$y\left(x\right)=\mathbf{ReLU}\left(x\right)=max(0,x).$$

(6)

$gap\left(\mathcal{F}\right)$ in Equation (1) is global average pooling of feature $\mathcal{F}$:

$$gap\left(\mathcal{F}\right)=\frac{1}{H\times W}\sum _{h=1}^H\sum _{w=1}^W{\mathcal{F}}_{[:,h,w]}.$$

(7)

LCC of $\mathcal{F}$ is defined as below:

$$\mathbf{LCC}\left(\mathcal{F}\right)=\mathbf{BN}\left(PWCon{v}_2\left(\mathbf{ReLU}\left(\mathbf{BN}\left(PWCon{v}_1\left(\mathbf{F}\right)\right)\right)\right)\right),$$

(8)

where $PWConv$ is pointwise convolution that uses a $1\times 1$ kernel to aggregate channel context for each spatial position.

With $\mathbf{GFC}\left(\mathcal{F}\right)$ and $\mathbf{LCC}\left(\mathcal{F}\right)$ in Equations (1) and (8), the attentional weight ($\mathbf{AW}$) of feature $\mathcal{F}$ in Figure 5 is defined as:

$$\mathbf{AW}\left(\mathcal{F}\right)=\mathbf{GFC}\left(\mathcal{F}\right)\oplus \mathbf{LCC}\left(\mathcal{F}\right),$$

(9)

where ⊕ is the broadcasting addition that adds scalars to higher-dimensional tensors. Lastly, as shown in the upper part of Figure 5, AFF is defined as:

$${\mathcal{F}}_{fused}=\mathbf{AFF}({\mathcal{F}}_1,{\mathcal{F}}_2)=\mathbf{AW}({\mathcal{F}}_1\oplus {\mathcal{F}}_2)\otimes {\mathcal{F}}_1+(1-\mathbf{AW}({\mathcal{F}}_1\oplus {\mathcal{F}}_2))\otimes {\mathcal{F}}_2,$$

(10)

where ⊕ is the same broadcasting addition as that in Equation (9), and ⊗ is the element-wise multiplication that adds corresponding elements between tensors.

The second part of the AFF-embedded CNLA block is the CNLA block that is calculated from the output of the AFF block ${\mathcal{F}}_{fused}$. The CNLA block is based on an improved nonlocal (NL) block [29]. In [29], the NL operation was defined as:

$${\mathbf{y}}_i=\frac{1}{\mathcal{C}\left(x\right)}\sum _{\forall j}f({\mathbf{x}}_i,{\mathbf{x}}_j)g\left({\mathbf{x}}_j\right),$$

(11)

where i is the position index of the output, and j enumerates all possible positions. $\mathbf{x}$ and $\mathbf{y}$ are input and output signals, respectively, with the same size. The $f({\mathbf{x}}_i,{\mathbf{x}}_j)$ function generates a scalar value between i and all j, and it is discussed in the next section. Function $g\left({\mathbf{x}}_j\right)$ generates a value for position i, and it can be considered in the form of linear embedding: $g\left({\mathbf{x}}_j\right)=W_g{\mathbf{x}}_j$, where $W_g$ is a learnable parameter. $\mathcal{C}\left(\mathbf{x}\right)$ is the normalization coefficient. With the NL operation that is defined in Equation (11), the nonlocal block is defined as:

$${\mathbf{z}}_i=W_z{\mathbf{y}}_i+{\mathbf{x}}_i$$

(12)

where ${\mathbf{y}}_i$ is defined in Equation (11), $+{\mathbf{x}}_i$ denotes a residual connection, and $W_z$ is a learnable parameter.

The function of $f({\mathbf{x}}_i,{\mathbf{x}}_j)$ in Equation (11) has multiple potential options. In [24], cosine similarity was introduced as the $f({\mathbf{x}}_i,{\mathbf{x}}_j)$ function into the CNLA block:

$$f({\mathbf{x}}_i,{\mathbf{x}}_j)=\frac{{\mathbf{x}}_i^T{\mathbf{x}}_j}{\parallel {\mathbf{x}}_i\parallel \parallel {\mathbf{x}}_j\parallel }.$$

(13)

Lastly, output ${\mathbf{y}}_j$ of CNLA is defined as:

$${\mathbf{y}}_j=\sum _{\forall j}{s}_{i,j}g\left({\mathbf{x}}_j\right)+{\mathbf{x}}_j,$$

(14)

where $s_{i,j}$ is the $softmax$ operation performed on a row of the similarity map:

$$s_{i,j}=\frac{expf({\mathbf{x}}_i,{\mathbf{x}}_j)}{{\sum }_{\forall j}expf({\mathbf{x}}_i,{\mathbf{x}}_j)}.$$

(15)

3.2. Region Proposal Module

Following FEM described in Section 3.1, the region proposal module (RPM) [17] is applied to estimate the rough location of objects. As shown in Figure 6, in the RPM, a feature map is attached to a $3\times 3$ convolutional layer to generate the intermediate feature map (IFM) ${\mathcal{F}}_{IFM}$, and it was designed to collect information from neighboring regions of the feature map. Referring to [17], k reference boxes, namely, anchors, were predefined with different aspect ratios in every location of the intermediate feature map. To obtain the rough position of chips, ${\mathcal{F}}_{IFM}$ is input into two $1\times 1$ convolutional layers to obtain the scoring and regression layers. The scoring layer had k channels, and the regression layer had $4k$ channels. For each location in the feature map, anchors with a higher score than the predefined thresholding were chosen as candidate ROIs, and accuracy could be further improved with the regression layer.

3.3. Detection Module

The detection module was designed to estimate the precise location and class of chips. Feature maps from FEM ${\mathcal{F}}_{feature}$ could be used in these tasks, but they could not provide feature distribution for the detection module. To solve this problem, the spatial attention block (SAB) [18] was applied to reweight ${\mathcal{F}}_{feature}$ with spatial dimensions from RPM. As shown in Figure 7, the intermediate feature map of FPM ${\mathcal{F}}_{IFM}$ was attached to a $1\times 1$ convolutional layer, followed by a BN layer and a sigmoid layer; then, it was multiplied by feature map ${\mathcal{F}}_{feature}$ to generate final result ${\mathcal{F}}_{SAM}$. The SAB is define as:

$${\mathcal{F}}_{SAB}={\mathcal{F}}_{feature}·sigmoid\left(\mathbf{BN}\left({\mathcal{F}}_{PRM}\right)\right),$$

(16)

where $\mathbf{BN}$ denotes the BN layer described in Equation (5), and $sigmoid$ is defined as:

$$sigmoid\left(x\right)=\frac{e^x}{e^x+1}.$$

(17)

As shown in Figure 3, all of four outputs of the feature maps from FEM were attached to SAM to generate spatial attentional feature maps $\{s_i,i\in [2\cdots 4]\}$. Although several measures were used in previous section, different information is contained in different feature maps. To combine these feature maps, as shown in Figure 8, the multiscale ROI fusion block [23] was introduced in our work. In the RPM section, ROIs were estimated in every feature map. Then, ROI information is input into the ROI align pooling (ROIAlign) layer [30] to extract ROI features that had the same size. In ROIAlign, ROIs are subdivided into spatial bins; the exact values of these bins are computed with bilinear interpolation and generate feature of ROI with aggregates. To obtain multiscale ROI features, these ROI features are fused with element-wise max operation.

Lastly, in order to detect the class and precise location of electronic chips, a fused feature is attached to a sequence of fully connected layers, followed by two branches of fully connect layers: one produces scores about k object classes, and the other generates four values for each K class that encodes refined bounding-box information.

3.4. Multitask Loss

Our detection network was assigned two tasks: classify and regress the bounding box that corresponded to two branches in the detection module. The loss function of our network is defined as follows [17]:

$$\mathcal{L}(\left\{p_i\right\},\left\{t_i\right\})=\frac{1}{N_{cls}}\sum _i{\mathcal{L}}_{cls}(p_i,p_i^{\ast })+\lambda \frac{1}{N_{reg}}\sum _i{p}_i^{\ast }{\mathcal{L}}_{reg}(t_i,t_i^{\ast }),$$

(18)

where i is the index of anchors, and $p_i$ is the probability of the object in anchor i. The two terms are normalized by $N_{cls}$, $N_{reg}$, which are the minibatch size and the number of anchor locations, respectively, and they are weighted by a balancing parameter $\lambda$. ${\mathcal{L}}_{cls}(p_i,p_i^{\ast })$ and ${\mathcal{L}}_{reg}(t_i,t_i^{\ast })$ in Equation (18) are described as follows.

In Equation (18), $p_i^{\ast }$ is the ground truth of $p_i$, defined as:

$$p_i^{\ast }=\left\{\begin{array}{cc}1\hfill & \mathrm{object}\mathrm{is}\mathrm{in}\mathrm{the}\mathrm{anchor}i,\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(19)

Classification loss ${\mathcal{L}}_{cls}$ is log loss over two classes (object versus not object).

$${\mathcal{L}}_{cls}(p_i,p_i^{\ast })=-\log [p_i{p}_i^{\ast }+(1-p_i)(1-p_i^{\ast })].$$

(20)

$t_i$ is the reg vector with 4 elements that represents the predicted bounding box:

$$\left\{\begin{array}{cc}{t}_x=\frac{x-x_a}{w_a},\hfill & t_y=\frac{y-y_a}{h_a},\hfill \\ t_w=\frac{w}{w_a},\hfill & t_h=\frac{h}{h_a}.\hfill \end{array}\right.$$

(21)

$t_i^{\ast }$ is the ground truth of $t_i$:

$$\left\{\begin{array}{cc}{t}_x^{\ast }=\frac{x^{\ast }-x_a}{w_a},\hfill & t_y^{\ast }=\frac{y^{\ast }-y_a}{h_a},\hfill \\ t_w^{\ast }=\frac{w^{\ast }}{w_a},\hfill & t_h^{\ast }=\frac{h^{\ast }}{h_a},\hfill \end{array}\right.$$

(22)

where $x, y, w, \mathrm{and} h$ denote the central coordinates, width, and height of the predicted bounding box. To improve robustness, notation ${\mathcal{L}}_{reg}$ with $L1$ smooth is defined as:

$${\mathcal{L}}_{reg}(t_i,t_i^{\ast })=\left\{\begin{array}{cc}0.5{(t_i-t_i^{\ast })}^2\hfill & if|t_i-t_i^{\ast }|<1,\hfill \\ |t_i-t_i^{\ast }|-0.5\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(23)

With loss function $\mathcal{L}$ defined in Equation (18), learnable parameters can be trained with the stochastic gradient descent method:

$${\theta }^{(t+1)}={\theta }^{\left(t\right)}-\gamma \frac{\partial \mathcal{L}}{\partial \theta },$$

(24)

where $\theta$ is one of learnable parameters, and $\gamma$ is the learning rate.

4. Experimental Results

4.1. Dataset

In this section, the proposed chip detection method is evaluated on our chip image dataset. In the electronic industry, the appearance of PCBs and chips largely varies. So, it is impossible to generate a unified dataset that meets the requirements of all chip detection applications. To this end, chip detection application generally relies on small-scale datasets. This image dataset had 367 images, and was divided into two subsets: the training dataset contained 330 images, and the evaluation dataset contained 37 images. Images of the dataset were captured by the electronic chip detection system that is illustrated in Figure 1. On the basis of the image capture strategy, chip images were randomly cropped, and their width was between 300 and 700 pixels. Each image contained at least five chips belonging to at least two classes. Distribution of instances of the chip dataset is shown in

Table 2. Distribution of instances.

Dataset	Resistor	Capacitors	Transistor	IC	Inductor	Total
Training	665	670	307	183	54	1879
Evaluation	70	62	32	16	8	188

.

4.2. Implement Details

The proposed chip detection method was evaluated on a workstation with an Intel Xeon (R) Gold 6278C CPU and a Nvidia Tesla V100 GPU. The network was implemented with Python programming language based on PyTorch [31] and its expansion pack Detectron2 [32]. The pretrained checkpoint of ResNet from Detectron2 was used to initialize the backbone of our method.

As discussed in Section 4.1, electronic chip detection methods are always trained with small-scale datasets and are prone to overfitting. To solve this problem, data augmentation was applied to our method. Through expanding the training dataset, data augmentation technology is able to improve generalization and robustness against changes in the input image, such as regarding image density, object position, and object orientation. In this paper, three augmentations are used:

(1)

random crop augmentation: cropping a region with random size from raw images;

(2)

random flip augmentation: randomly flipping the image;

(3)

small object augmentation [33]: copying small objects from the original position and pasting them to different positions.

To learn parameters for our method, a back-propagation-based optimization method is applied to minimize the loss defined in Equation (18), which is a function of weight parameters. First, the derivative of the loss function to each weight was calculated; then, a stochastic gradient descent method with momentum was applied to update weights in the direction of the fastest gradient decent until the maximal iteration. The previous momentum was used to accelerate the current gradient: update direction was defined by the previous update direction and the gradient of the current batch. In other words, if current and previous gradient directions are the same, update speed is higher; otherwise, update speed is lower. In our work, the learning rate was set to 0.0001, momentum was set to 0.9, and the maximal iteration was to 40,000.

In the RPM, the area of anchors for every pyramid feature maps was assigned to $\{{32}^2,{64}^2,{128}^2,{256}^2,{512}^2\}$, and the aspect ratio of all anchors was $[\frac{1}{3},\frac{1}{2},1,2,3]$. The maximal iteration was set to 8000, and the loss curve in training our method is shown in Figure 9. Experimental results of our method are demonstrated in Figure 10.

4.3. Evaluation Metrics

The performance of our method was established with the PASCAL criteria [34]. First, detection results were sorted by their confidence scores; then, the IoU was calculated for these results:

$$IoU=\frac{DetectionResult\cup GroundTruth}{DetectionResult\cap GroundTruth},$$

(25)

where $DetectionResult$ is the bounding box of the detection result, and $GroundTruth$ is the annotation box. Then, $l_i^t$ is defined as:

$$l_i^t=\left\{\begin{array}{cc}1\hfill & a_i>t\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

(26)

where $a_i$ is the IoU of i-th detection result, and t is the threshold. Precision p and recall r are defined as follows:

$$\begin{array}{cc}{p}_i^t\hfill & =\frac{t{p}_i^t}{t{p}_i^t+f{p}_i^t}\hfill \\ r_i^t\hfill & =\frac{t{p}_i^t}{n_p^t}\hfill \end{array}$$

(27)

where $n_p^t$ is number of positive samples, and $t{p}_i^t$ and $f{p}_i^t$ are true positive and false positive, respectively:

$$\begin{array}{cc}t{p}_i^t\hfill & =t{p}_{i-1}^t+l_i^t\hfill \\ f{p}_i^t\hfill & =t{p}_{i-1}^t+1-l_i^t\hfill \end{array}$$

(28)

On the basis of the area under precision recall curve, the $AP$ was calculated as follows:

$$A{P}^t=\sum _i^{n_p}{p}_i\Delta r.$$

(29)

The final $mAP$ was calculated with the average value of $AP$ for N classes:

$$mA{P}^t=\frac{1}{N}\sum _{j=1}^{N}A{P}_i^t.$$

(30)

On the basis of $mAP$ defined above, alternative criteria $mA{P}^{coco}$ were calculated by the average value of $AP$ with $t=$ 0.05:0.05:0.95 [35,36].

4.4. Evaluation Results

Table 3. $mA{P}^{0.5}$, $mA{P}^{0.75}$, and $mA{P}^{coco}$ on testing dataset.

	${\mathit{mAP}}^{0.5}$	${\mathit{mAP}}^{0.75}$	${\mathit{mAP}}^{\mathit{COCO}}$
Faster R-CNN	0.96570	0.91685	0.76109
Our method	0.98745	0.95142	0.81130

shows the accuracy gap between our method and the faster R-CNN method, and

Table 4. $mA{P}^{coco}$ on testing dataset and per category of bounding box $mA{P}^{coco}$.

	${\mathit{mAP}}^{\mathit{COCO}}$	Resistor	Capacitors	Transistor	IC	Inductor
Faster R-CNN	0.76109	0.73493	0.77381	0.75677	0.81387	0.72608
Our method	0.81130	0.72234	0.80324	0.77550	0.88759	0.86782

shows the difference between these two methods by category. As shown in Table 3, our method achieved promising results, outperformomg benchmark method R-CNN.

As shown in Table 3, the results of $mA{P}^{0.5}$ and $mA{P}^{0.75}$ in our method were $0.98745$ and $0.95142$, respectively, while $mA{P}^{COCO}$ was only $0.81130$. According to the definitions, $mA{P}^{0.5}$ evaluates detection results with the IoU threshold between bounding boxes of the detection results and ground truth, and this was equal to $50\%$. $mA{P}^{0.75}$ evaluates detection results with a threshold equal to $75\%$; and $mA{P}^{COCO}$ combines the IoU threshold from $50\%$ to $95\%$ in intervals of $5\%$, that is, the requirement for IoU is higher. Therefore, our method could roughly detect electronic chips from the image, but, as shown in Figure 10, there was a certain error in the central coordinates and the size area. Hence, the accuracy of the bounding box could be further improved.

As shown in Table 4, the $mA{P}^{COCO}$ the results of our method in detecting capacitors, transistors, ICs, and inductors were $0.80324$, $0.77550$, $0.88759$ and $0.86782$, respectively, which ertr lower in detection resistors than those in the faster R-CNN method. According to our analysis, this result was because the surface of the resistors had text indicating the resistance value, so their surface texture was relatively complex. Compared with the faster R-CNN method, due to the extraction of more object features, our method was prone to overfitting when the amount of data was not large enough. Therefore, it is necessary to improve the accuracy of our method in detecting complex objects with a small amount of training data.

5. Conclusions

This paper proposed a novel electronic chip detection method that was trained with a small-scale chip dataset. Three aspects distinguish our work from previous works: first, our method was designed to detection chips that belong to different classes in complex backgrounds; second, AFF-embedded CNLA module and pyramid feature module were combined to extract features from chip images; third, pyramid feature maps were enhanced with the region intermediate feature map to classify and locate chips. The experiment showed that our work outperformed a landmark method. There are two challenges for our work: (1) the accuracy of the bounding box needs to be further improved; (2) the detection accuracy of objects with complex textures needs to be further improved. We will focus on improving the precision of the bounding boxes of electronic chips and the performance of the few-shot electronic chip detection method.