1. Introduction
Fiber-reinforced composites (FRC) have been widely used in various industrial fields, such as aerospace, marine, and sporting goods, thanks to their excellent fatigue and corrosion resistance, super strength, high toughness and light-weight properties [
1]. FRC is commonly fabricated by overlapping a certain number of fiber prepreg sheets on the mold with regularly alternating orientations layer-by-layer, as shown in
Figure 1. Fiber prepreg sheets are pre-impregnated fibers that have been coated with a resin matrix and then partially cured, making them ready for use in composite fabrication. The fibers are the primary load-carrying element in fiber prepreg sheets, and provide strength and stiffness in the direction of their alignment. Thus, the fiber orientations have a significant impact on the mechanical properties of the final composite parts. However, the automated or manual fiber prepreg sheets placement could bring the discrepancies between the intended fiber orientation and the fiber actual orientation, which may result in the overall system performance degradation. For instance, the deviation of
can reduce the strength of FRC components by almost 20% [
2]. Therefore, an efficient fiber orientation inspection system for FRC fabrication is desperately needed if the fiber placement system is not under good control.
Fiber orientation inspection techniques can be broadly classified into two categories: destructive testing and non-destructive testing (NDT) [
3]. The destructive testing techniques are typically used in research and development. It focuses on understanding the mechanical properties of the material rather than on manufacturing quality control [
4]. NDT includes a range of techniques, such as automated visual inspection [
5], ultrasonic testing [
6], X-ray radiography [
7], and thermography [
8,
9,
10], which allow for the inspection of the fiber orientation without damaging the material. Automated visual inspection uses image processing algorithms to analyze the texture images of FRC and determines the fiber orientation. Compared to other NDT techniques, automated visual inspection can utilize the common 2D camera. It exerts great potentials in measuring fiber orientation, thanks to its advantages in cost-effectiveness, computational efficiency and accuracy.
The Hough transform (HT) and the deep Hough Transform (DHT) are major image processing methods for automated visual inspection of fiber orientation [
11,
12], as the “line-like” structures of the fiber texture in FRC are readily detectable using these methods [
13]. The HT works by mapping image points to a parameter space in which a line can be represented as a single point. In the parameter space, a simple voting scheme is used to identify the most likely line parameters. The DHT employs a deep neural network to perform the mapping from image features to parameter space, which can improve the accuracy, efficiency, and robustness of the HT [
11]. However, the DHT still suffers two challenging issues in measuring fiber orientation, which are listed as follows.
(1) Sensitivity to background anomalies: Background anomalies caused by mold regularly appear in FRC images, as shown in
Figure 2. The mold background may cause dark spots, lines, or speckles on the FRC images, which could be mixed with fibers and cause the DHT to detect false lines in the image. The reduction of FRC regions in the image also generates more short line segments, further decreasing the line detection performance of DHT.
(2) Sensitivity to longline segments anomalies: The DHT detects a line by searching group of points in an image that lie on the line. Longer line segments containing more points are assigned higher weights in Hough space. However, there are often many longline segments anomalies on the surface of FRC, such as reference line emitted by a laser projector, or gap caused by FRC fracture, as shown in
Figure 2. To provide a visible marker for aligning the FRC, the reference line is projected onto its surface. FRC can become damaged or worn over time, which can lead to gaps on the surface. These longline segments anomalies can gain more votes in the Hough space than shorter, true “line-like” structures, which can seriously affect the performance of fiber orientation measurement. Traditional Hough normalization can improve the robustness of the traditional Hough transform to longline segments anomalies [
14], but also makes it more sensitive to background anomalies. Because there are many short line segments near the edges of the FRC regions in the images with background anomalies. After traditional Hough normalization, these short line segments may produce peaks in the parameter space that exceed the detection threshold [
15], thus reducing the performance of the model in detecting true “line-like” structures.
To address these issues, we propose an attention-based deep Hough network with Hough normalization, which is robust to the background and longline segments anomalies. The main contributions of this paper are summarized as follows.
(1) An attention-based deep Hough network (DHN) is proposed to remove background anomalies caused by mold in FRC images, which seamlessly integrates the attention network and the Hough network together. The attention network is used to learn the important regions of the FRC image that contain the fiber orientation information, while the Hough network is used to detect the fiber orientations in those regions.
(2) The deep Hough normalization is proposed to improve the accuracy and robustness of the DHT when applied to FRC images with longline segments anomalies and background anomalies. It works by normalizing the accumulated votes in the deep Hough space by the length of the corresponding line segment, which completely eliminates the influence of line length on DHT. The deep Hough normalization can make the DHT robust to both longline segments anomalies and background anomalies.
(3) Three separate datasets have been established, which include normal FRC images, FRC images with longline segments anomalies, and FRC images with background anomalies. Our method has been extensively evaluated on these datasets, and achieves the competitive performance against the state-of-the-art in F1, MAE and RMSE.
The rest of this paper is organized as follows:
Section 2 summarizes the related works.
Section 3 describes the materials and the proposed methods with sufficient detail.
Section 4 presents the experimental details and systematic analysis.
Section 5 concludes the whole paper.
3. Materials and Methods
3.1. Composite Specimen and Image Acquisition
The fiber prepreg sheet used in this study is CYCOM X850®(Cytec Industries, Woodland Park, NJ, USA). The CYCOM X850 is composed of the thermoplastics toughened epoxy and unidirectional T800 fibers. The resin content of the CYCOM X850 is 35% by weight and the fabric area density is 292 g/m.
To thoroughly evaluate the robustness of the proposed method in measuring fiber orientation under different real-world scenarios, we establish a normal dataset and two anomalous datasets, each for one type of anomalies (longline segments anomalies, background anomalies). Each dataset has 5400 training images, 3600 validation images, and 5400 test images.
The normal dataset contains 14,400 normal FRC images with 180 different fiber orientations ranging from 0° to 179° at 1° increments, and 80 types of illuminations which cover a large range of light conditions. In order to collect the normal dataset, a data collection platform has been built. The FRC data collection platform consists of four parts: camera, symmetric light source, computer, and two pieces of fiber prepreg sheets. The camera is a 2D camera (HikVision MV-CA050-12GC2) with lens (HikVision MVL-KF2524M-25MP, focal length: 25 mm), connected to the laptop by USB cable. The laptop is equipped with Intel Core i7 processing unit and 16G memory. Different light intensities can be obtained by adjusting the light source height, illumination angle, and light intensities. The ranges of light source heights relative to the FRC layer are [10 cm, 20 cm, 30 cm, 40 cm, 48.5 cm], and the ranges of the illumination angles are [20°, 45°, 75°, 90°]. The acquired images are RGB images with resolution of 2048 × 2048. After being converted to grayscale images, the acquired images are resized to 200 × 200 to compose the normal dataset.
It is well-known that training neural networks can be a tedious task, due to the high cost of obtaining the large amount of labeled data required to train a general model [
28]. Utilizing synthetic data for model training has become more popular in recent years as a way to address the issue, which can be obtained quickly and at much lower cost [
34]. Both the longline dataset and the background dataset contain a large number of synthetic images generated from the normal dataset.
The longline dataset contains FRC images with longline segments anomalies, such as reference lines emitted by a laser projector or gaps resulting from FRC fractures. The reference lines serve as a precise alignment of the fiber orientation in the actual lay-up scene, and FRC typically fractures along the direction of the fibers. Thus, the orientations of the reference lines or gaps should not deviate too much from the fiber orientation. On the longline dataset, the orientation of the reference lines and gaps deviates no more than ±10° from the fiber orientation. The process of simulating a gap resulting from FRC fractures involves drawing two parallel straight lines on the normal FRC image, filling the space between the lines with white to create a gap, and normalizing the gap to match the brightness of the surrounding FRC material. The gap widths are between 2 to 10 pixels.
The background dataset comprises FRC images with background anomalies caused by mold, which may create dark spots, lines, or speckles on the FRC images. The mold used in this study is a matte metal mold. To create a FRC image with background anomalies, the first step is to generate a binary mask, which can be divided into two parts. Then, we fill one part of the mask with the normal FRC image and the other part with the mold image. Finally, the mold image part is normalized to match the brightness of the FRC image, which ensures that the final synthetic image has a uniform level of illumination across both the FRC and mold regions. Two algorithms are employed to generate the binary mask, which include 1D random walks and B-splines. The area of the normal FRC part accounts for 20% to 40% of the total image area.
3.2. Preliminaries
The texture of FRC image consists of long parallel lines. The fiber orientation measurement for the FRC image is to detect the orientation of these parallel lines. As shown in
Figure 3, given a 2D image
, where
H and
W are the spatial size, we set the center of the image as the origin. A straight line
l is one representative fiber texture, which satisfies the following equation in the polar coordinate frame:
where
is the distance from the origin to the line
l,
is the angle between the normal of the line
l and x-axis. We define
as the fiber orientation of the FRC image.
As described in Equation (
1), every straight line in the image can be represented by two parameters
and
r. The two parameters are quantized into discrete bins. The discrete binned parameter space is denoted as the Hough space. We define the quantization interval for
r and
as
and
, respectively. Then the Hough space size, denoted with
and
R can be formulated below:
3.3. Attention-Based Normalized Deep Hough Network
The attention-based normalized Deep Hough Network (AttNorm-DHN) mainly contains four components: (1) an encoder network that extracts pixel-wise deep representations; (2) an attention network that learns the important regions of the FRC image which contains the fiber orientation information; (3) the Norm-DHT that converts the deep representations from the spatial space to the Hough space; (4) the orientation measurement module that measures the fiber orientation in the Hough space. The AttNorm-DHN is an end-to-end framework, which allows the network to learn the entire process directly from raw inputs to the desired outputs. During the training process, we optimize all the components from scratch without any pre-training or transfer learning. The pipeline of AttNorm-DHN is shown in
Figure 4.
3.3.1. Encoder Network
The encoder network is an important component that is responsible for extracting pixel-wise deep representations from the input image. The deep representations obtained from the encoder network have a higher level of abstraction compared to the raw pixel values of the input image. These representations are then used to capture the structural information of the image, which is crucial for accurately measuring the fiber orientation in FRC images.
Given an input image , the deep CNN features is firstly extracted with the encoder network, where C indicates the number of channels and H, W are the spatial size.
To perform a fine-grained deep Hough transform on the deep CNN features
X, it is necessary to ensure that the feature resolution of
X is sufficiently high. To achieve this, a shallow encoder network is designed.
Table 1 shows the network architecture of encoder network. The network mainly consists of multiple
convolution layers except for the last Conv
layer, which compresses the output from 128 dimensions to 64 dimensions in order to obtain more compact feature representations. We use ReLU activation in all layers of the encoder network [
35]. The Maxpool operation correspond to
maxpooling, which reduces the spatial resolution of the feature maps by a factor of 2 [
36].
3.3.2. Attention Network
The attention network takes the FRC image as input and produces an attention map. The attention map is a weight map indicating the importance of each spatial location in the output features of encoder network. It highlights the regions of the image that are most relevant for predicting the fiber orientation.
Given an input image
I, the attention weights
of input image is generated with the attention network. Then the attention weights
A is used to compute the weighted deep CNN features
.
where ⊗ indicates the element-wise multiplication.
Table 2 shows the network architecture of attention network. The network architecture starts with two
convolutional layers with ReLU activation functions. A maxpooling layer is then used to reduce the spatial resolution of the feature maps by a factor of 2, producing 64 feature maps of size
.
After that, a bottleneck-like network [
37] is added on top of it to obtain the attention weights. Specifically, the bottleneck-like network consists of four sets of convolutional layers, each comprising two
convolutional layers with ReLU activation functions and a maxpooling or upsampling layer. The upsampling operations correspond to
element replication, which increases the spatial resolution of the feature maps by a factor of 2. The first two sets of convolutional layers utilize maxpooling layers to gradually reduce the spatial resolution and increase the number of feature maps. The last two sets of convolutional layers employ upsampling layers to gradually increase the spatial resolution and reduce the number of feature maps.
Finally, the attention network utilizes two convolutional layers with ReLU activation functions and a convolutional layer with a Sigmoid activation function to obtain the attention weights.
3.3.3. Feature Transformation with Normalized Deep Hough Transform
The deep Hough transform (DHT) can transform lines in the image space into more compact representations in the parameter space [
11], which makes the lines easier to detect by neural network. The normalized deep Hough Transform (Norm-DHT) is an improved DHT, which takes advantages of the deep Hough Normalization to further improve the accuracy and robustness of the DHT in the presence of longline segments anomalies and background anomalies.
The Norm-DHT takes
as an input and produces the normalized deep Hough features
, where
and
R are the parameters in the Hough space, as described in Equation (
2). Different from the traditional Hough transform, DHT is used to vote in the deep spatial space. Each bin in the deep Hough space corresponds to the features along a straight line in the deep CNN features
X. Features of all pixels along line
l in
X are aggregated to the corresponding position
in
Y:
where
i is the position index.
and
are the parameters of line
l, according to Equation (
1).
Similar to traditional Hough normalization [
14], the deep Hough normalization is typically implemented as an additional step following the DHT. After the deep Hough features
Y is generated, the length of each line segment in the deep CNN features
X is calculated. Each bin in the deep Hough features
Y are then divided by the length of the corresponding line segment to perform Hough normalization, as shown in
Figure 5.
The length matrix
can be computed by applying the DHT to an all-ones matrix
with every pixel set at unity.
where
is the length of the line
l.
and
are the parameters of line
l.
Afterwards, the normalized deep Hough features
is obtained by dividing deep Hough features
Y with the length matrix
L.
At last, two 3 × 3 convolutional layers with ReLU activation functions and a 1 × 1 convolutional layer with a Sigmoid activation function are applied to yield the value in the Hough space
. The network architecture of Norm-DHT is shown in
Table 3.
3.3.4. Orientation Measurement Module
In this work, we need to measure the orientation
in the final Hough space regardless of the parameter
r. A simple method is employed to get the orientation vector from the final Hough space. The orientation vector
is obtained by accumulating the final Hough space
along the direction of
R:
where
i is the positional index along the direction of
R.
3.3.5. Loss Function
Note that we discretize the Hough space and obtain the orientation vector by accumulating Hough space along the direction of
R. Thus, the orientation measurement could be converted to the classification, i.e., to predict a probability for each orientation bin. This classification problem can be addressed with the standard cross-entropy loss to simply predict each orientation class independently:
where
m is the number of training samples.
is the predicted orientation vector of sample
i, and
is the ground-truth orientation vector of sample
i. We train the end-to-end AttNorm-DHN model by minimizing the loss
L via the standard Adam optimizer [
38].
5. Conclusions and Future Works
In this paper, the deep Hough normalization has been proposed for fiber orientation inspection. It normalizes the accumulated votes in the Hough space by the length of the corresponding line segment, making it easier for DHN to detect short, true “line-like” structures. In addition, we have designed an attention-based DHN that integrates attention and Hough networks to remove background anomalies caused by mold in FRC images. Finally, three datasets have been established to evaluate the proposed methods, and our proposed methods have been evaluated extensively on them. The experimental results and analysis revealed that the proposed method achieves the competitive performance against previous arts in terms of F-measure, MAE and RMSE in real-world scenarios with various types of anomalies.
There are several FRCs need further investigation to fully explore the applicability and robustness of the proposed method in different scenarios, including unidirectional fibers with waviness or crimp, transparent (glass) fibers, and woven fabrics with voids and cross-overs in them. These studies aim to enhance the method’s robustness and applicability across various materials, contributing to the development of high-performance composites and advanced materials.