PCBSNet: A Pure Convolutional Bilateral Segmentation Network for Real-Time Natural Scene Text Detection

Abstract

Scene text detection is a fundamental research work in the field of image processing and has extensive application value. Segmentation-based methods have time-consuming feature processing, while post-processing algorithms are excellent. Real-time semantic segmentation methods use lightweight backbone networks for feature extraction and aggregation but lack effective post-processing methods. The pure convolutional network improves model performance by changing key components. Combining the advantages of three types of methods, we propose a Pure Convolutional Bilateral Segmentation Network (PCBSNet) for real-time natural scene text detection. First, we constructed a bilateral feature extraction backbone network to significantly improve detection speed. The low extraction detail branch captures spatial information, while the efficient semantic extraction branch accurately captures semantic features through a series of micro designs. Second, we built an efficient attention aggregation module to guide the efficient and adaptive aggregation of features from the two branches. The fused feature map undergoes feature enhancement to obtain more accurate and reliable feature representation. Finally, we used differentiable binarization post-processing to construct text instance boundaries. To evaluate the effectiveness of the proposed model, we compared it with mainstream lightweight models on three datasets: ICDAR2015, MSRA-TD500, and CTW1500. The F-measure scores were 82.9%, 82.8%, and 78.9%, respectively, and the FPS were 59.1, 94.3, and 75.5 frames per second. We also conducted extensive ablation experiments on the ICDAR2015 dataset to validate the rationality of the proposed improvements. The obtained results indicate that the proposed model significantly improves inference speed while enhancing accuracy and demonstrates good competitiveness compared to other advanced detection methods. However, when faced with curved text, the detection performance of PCBSNet needs to be improved.

1. Introduction

Object detection is an important research direction in the field of computer vision, aiming to identify and locate different objects in images. In practical applications, object detection tasks face various challenges, such as vehicle trajectory extraction [1], single-object perception [2], precision agriculture detection [3], and scene text detection. Among these, scene text detection is a fundamental problem in the field of object detection and can be applied to many real-world tasks, such as license plate recognition [4], scene analysis [5], and autonomous driving [6]. The accuracy and real-time performance of scene text detection is of significant importance for the subsequent tasks of text recognition and understanding of a scene.

In recent years, significant progress has been made in scene text detection with the development of Convolutional Neural Networks (CNNs) [7]. Researchers have proposed various scene text detection methods based on CNNs, including regression-based and segmentation-based approaches. Regression-based scene text detection directly regresses text instance boundaries based on object detection methods [8], which enables fast detection and avoids error accumulation in multiple stages. However, this method has limited text representation forms (axis-aligned rectangles or rotated rectangles) and weak generalization capability. It struggles to produce satisfactory results when detecting arbitrary-shaped text, which greatly affects the performance of subsequent text recognition tasks. On the other hand, segmentation-based scene text detection locates text instances by pixel-level classification, allowing for accurate representation of arbitrarily shaped text. It has become the mainstream method in scene text detection. However, segmentation-based methods face challenges in complex post-processing and inefficient feature processing, making it difficult to simultaneously achieve high accuracy and inference speed.

The complexity of post-processing arises from the fact that most post-processing algorithms prioritize accuracy over speed. To address this issue, DBNet (Differentiable Binarization Network) [9] introduces differentiable binarization, replacing binary operations with approximate step functions. It incorporates this approach into a trainable segmented network during training to improve the slow inference speed of post-processing.

The reason for the inefficient feature processing methods is that segmentation-based methods are typically designed for general object detection, where high accuracy heavily relies on the performance of the backbone network for feature processing, often neglecting computational costs. For example, CRAFT [10] introduces additional computations on high-resolution feature maps using U-Net [11], resulting in reduced model speed. Pixel-Anchor [12] uses dilated convolutions [13] to remove downsampling operations, leading to high memory usage and computational complexity. ABCNet [14] employs Residual Networks (ResNets) [15] that require substantial computational resources for training and inference when the network structure is deep. TextFuseNet [16] utilizes Feature Pyramid Networks (FPNs) [17] to fuse multi-scale features in a bottom-up manner, incurring significant overhead.

In order to balance the accuracy and inference speed of the scene text detection model while meeting the requirements of real-time applications, we have constructed a Pure Convolutional Bilateral Segmentation Network (PCBSNet). The most critical issue we addressed was how to overcome the inefficient feature processing methods of existing approaches while avoiding accuracy loss.

The post-processing part of DBNet [9] maintains a high inference speed, but the feature extraction and fusion methods of ResNet + FPN are time-consuming. Considering that a lightweight backbone can significantly improve the feature extraction speed, but the accuracy inevitably decreases due to the fewer network layers, we introduce the design structure of ConvNeXt [18] into the backbone network of BiseNet V2 [19] to construct the Bilateral Feature Extraction (BFE) module, replacing the feature extraction method of ResNet. This significantly improves the inference speed while minimizing accuracy loss. The feature fusion part of BiseNet V2 [19] adopts a simple Bilateral Guide Aggregation (BGA) module, ignoring the importance difference between the two branch features. Therefore, we embedded the Triplet Attention Fusion (TAF) [20] and Triple Channel-Spatial Attention (TCSA) modules into the BGA module to construct the Efficient Attention Aggregation (EAA) module, allowing the network to focus more on important semantic information. We also replaced DWConv (Depthwise Convolution) in the BGA module with the faster Pconv (Partial Convolution) to further enhance our real-time requirements. Finally, we utilized Coordinate Attention (CA) [21] to design the Feature Strengthen (FS) module, which aggregates input features in the horizontal and vertical directions separately to capture position-sensitive information and channel long-range dependencies, compensating for the semantic bias in the fusion process. To summarize, the main contributions of this paper are as follows:

• We abandoned the inefficient feature extraction method of DBNet and used an improved Bilateral Feature Extraction module as a replacement for ResNet18 in the feature extraction part of PCBSNet. This improves the efficiency of feature capture and enables the implementation of a lightweight network.
• We embedded attention mechanisms in the fusion part of BiSeNet V2, creating the Efficient Attention Aggregation module. This module guides the network to efficiently and adaptively fuse both detail information and semantic information.
• We designed a Feature Enhancement module for the fused features to obtain more accurate and reliable feature representations.
• We evaluated our proposed PCBSNet for real-time scene text detection on the ICDAR2015, MSRA-TD500, and CTW1500 datasets, comparing it with existing mainstream lightweight methods. The experimental results demonstrate that our approach achieves a good balance between accuracy and real-time performance.

The rest of the paper is organized as follows. Section 2 provides an overview of the related work in scene text detection and the techniques used in our research. Section 3 presents a detailed description of our proposed real-time scene text detection network, PCBSNet. Section 4 presents extensive experiments and provides a detailed analysis of the results. Section 5 concludes the paper, summarizes the findings, and suggests future research directions.

2. Related Work

Semantic segmentation-based methods have become one of the mainstream approaches for scene text detection. These methods extract features using a backbone network to obtain pixel-level label predictions and employ post-processing algorithms to aggregate pixels for text detection. For instance, PAN (Pixel Aggregation Network) [22] utilizes a ResNet backbone network to extract features and introduces the Feature Pyramid Enhancement Module (FPEM) and the Feature Fusion Module (FFM) to improve feature fusion accuracy. The Pixel Aggregation (PA) post-processing method is used to aggregate text pixels. However, simple pixel aggregation may struggle with detecting adjacent text instances. To address this, PSENet (Progressive Scale Expansion Network) [23] employs ResNet + FPN backbone network feature extraction and proposes a progressive scale expansion post-processing algorithm based on a breadth-first search to accurately separate adjacent text instances. These post-processing methods involving expansion strategies are complex and computationally expensive. To tackle this issue, DBNet [9] introduces differentiable binarization (DB), which trains the binarization process along with the segmentation network to significantly improve post-processing efficiency. However, this method struggles to handle text within text instances. Therefore, FCENet (Fourier Contour Embedding Networks) [24] utilizes compact Fourier feature vectors to approximate text contours and employs Fourier inverse transformation as a post-processing step to accurately depict text contours and achieve correct text detection within text instances. Although these algorithms have achieved good performance in scene text detection, their effectiveness highly relies on feature processing and post-processing procedures. Additionally, these methods employ deep networks for feature extraction, which, while achieving desired accuracy, often come with long inference times that hinder real-time applications.

Considering the importance of real-time performance in scene text detection, researchers have recently explored the use of real-time semantic segmentation approaches for efficient feature processing, aiming to significantly reduce inference time. Real-time semantic segmentation employs simple multi-branch networks to extract different semantic features from images, thereby improving feature processing efficiency. For example, BiSeNet V1 and V2 [19,25] adopt a dual-branch structure for feature capturing and design parallel context and spatial paths to balance segmentation accuracy and inference speed. STDC [26] removes the spatial path of BiSeNet V1 [25] and enhances the base model by incorporating detail prediction. BiSeNet V3 [27] also eliminates the spatial path of BiSeNet V1 [25] and applies traditional edge detection techniques in this field, introducing coordinate feature refinement and aggregation modules to achieve the best trade-off between speed and accuracy. However, these methods often suffer from incomplete feature extraction when the text information in the image is complex, leading to missed or false detections.

To address this limitation, enhancing the network structure becomes crucial. The idea of designing network structures to enhance model detection performance originated from the Vision Transformer (ViT) [28]. ViT replaces convolutional operations with self-attention mechanisms, enabling extensive information, interaction, and global context capturing. However, the large number of parameters in ViT makes it challenging to apply to real-time tasks. To overcome this, recent studies have attempted to simulate several key success factors of ViT using convolutional operations, including long-range modeling, dynamic weights, robustness to distortion, and higher-order spatial interactions. For example, GFNet [29] combines 2D-FFT (Fast Fourier Transform) and 2D-IFFT (Inverse Fast Fourier Transform) to dynamically learn long-range spatial dependencies. RepLKNet [30] employs a 31 × 31 super-large convolution kernel to increase the effective receptive field, and small convolution kernels for parameter reorganization to alleviate distortion. HorNet [31] designs the Recursive Gated Convolution (RGConv) to explicitly model high-order interactions between any two spatial positions. ConvNeXt [18] incorporates the micro-design of Swin Transformer [32] into convolutional networks to explore the limits achievable with purely convolutional networks. ConvNeXt V2 [33] proposed a fully convolutional masked autoencoder framework based on ConvNeXt [18] and introduced the Global Response Normalization (GRN) layer, achieving a remarkable accuracy of 76.7% on the ImageNet dataset.

Through an in-depth investigation of the aforementioned methods and by leveraging their respective advantages, we propose a PCBSNet for real-time scene text detection.

3. Methods

The structure of our proposed PCBSNet model is shown in Figure 1. Firstly, the image is input into the BFE module, and detailed and semantic information is captured through the LDEB and ESEB. The output feature maps of the two branches are adaptively aggregated through the EAA module, and the aggregated features enter the FE module to obtain more accurate and reliable feature expression. Finally, an approximate binary map is obtained through DB post-processing, which is our final prediction map.

3.1. Bilateral Feature Extraction

This section introduces the proposed BFE module. Traditional segmentation-based text detection methods often use ResNet [15] for feature extraction, which is inefficient and inadequate for real-time detection tasks. To address this issue, we integrated the design of ConvNeXt [18] into the bilateral feature extraction network of BiSeNet V2 [19]. We design the LDEB and ESEB to capture target contour details and semantic information, respectively. We replaced the ResNet [15] backbone with BFE, significantly reducing the network parameters and computational complexity, thus achieving a lightweight improvement of the network.

3.1.1. Low Detail Extraction Branch

The LDEB is responsible for extracting spatial information from the image for text detection. Low-level details play a crucial role in semantic segmentation, as they capture the relationships between each pixel and its neighboring pixels, as well as the relationships between each pixel and all other pixels in the entire image, forming the semantic information of the image.

In order to capture rich spatial information while maintaining high inference speed, we have designed the LDEB using a VGG [34] network structure instead of a memory-intensive residual connection structure. We stacked three layers of the VGG network, with channel numbers set as 64, 128, and 256, respectively, to ensure a larger spatial size and channel capacity.

For the input image IM, the detail feature $X^{*}$ is obtained through the LDEB. The calculation process is as follows:

$$X^{*}=LDEB\left(IM\right)$$

(1)

where $LDEB\left(·\right)$ represents the feature passing through the LDEB.

3.1.2. Efficient Semantic Extraction Branch

In parallel with the LDEB, the ESEB is designed to capture high-level semantics, which is characterized by a deep level, large receptive field, and low channel capacity. In the first stage, the Stem block is used to expand the receptive field using the fast downsampling strategy, and the structure is shown in Figure 2a. The final stage uses CE blocks, with global average pooling and residual connections effectively embedding global context information, as shown in Figure 2b. The remaining details of the middle layer will be explained in detail below.

With the breakthrough development of pure convolutional networks, it can be observed that the effectiveness of BiSeNet V2 [19] is attributed to its high-level semantic branch aggregation and expansion layer GE (Gathering Expansion Layer) similar to the inverted bottleneck structure of ConvNeXt [18], as shown in Figure 3a–c. Compared with the standard convolution network, the GE1 and GE2 blocks have larger receptive field by superimposing the convolution layer. A design similar to ConvNeXt [18] can provide convolutional networks with higher feature expression capabilities to capture advanced semantics. Obviously, the more efficient the semantic extraction branch and the ConvNeXt [18] structure are, the more sufficient the semantic extraction is, and the higher the accuracy of scene text detection.

Following the design principles of ConvNeXt [18], a pure convolutional bilateral (PCB) block was constructed on the basis of GE. The main improvements include:

(1)

Broader Channels: The original channel capacity of the pure convolutional network BiSeNet V2 [19] is (16, 32, 64, 128), which is one-fourth of ResNet18 (64, 128, 256, 512). In comparison, ConvNeXt [18] expands the channel numbers to (96, 192, 384, 768). Considering the potential misdetections and omissions of text instances, richer semantic information is needed. Therefore, PCBSNet enlarges the channel numbers (32, 64, 128, 256), which is one-third of ConvNeXt [18].

(2)

Agile Convolutions: DWConv [35] has a visible effect on reducing FLOPs, but replacing conventional convolutions with DWConv leads to a significant drop in accuracy. Research [36] has found that increasing the network width to compensate for accuracy typically results in higher memory access. PCBSNet employs PConv [37] with a ratio of 1/4 as a substitute for DWConv, optimizing the cost by leveraging feature map redundancy.

(3)

Larger Convolution Kernel Sizes: Since VGG, mainstream CNNs usually adopt smaller convolution kernels like 3 × 3 and 5 × 5. However, ConvNeXt [18] achieves saturated performance by using a 7 × 7 convolution. In the GE1 block of the BiSeNet V2 [19] network, a 5 × 5 receptive field is achieved by stacking two 3 × 3 convolutions, but it is still smaller than that of the ConvNeXt block. PCBSNet replaces the second 3 × 3 convolution in the stacked GE1 block with a 5 × 5 convolution, expanding the receptive field to 7 × 7, matching the receptive field of ConvNeXt [18].

(4)

More Suitable Activation Function: Most existing models attach a batch normalization layer (BN) and a ReLU (Rectified Linear Unit) layer after each convolutional layer. However, research [37] has shown that this model structure does not achieve the best performance for networks with smaller channel numbers. Considering the lightweight and high-performance aspects, PCBSNet uses GELU (Gaussian Error Linear Unit) as a replacement for ReLU to improve model convergence without increasing computational cost.

The improved PCB1 and PCB2 are shown in Figure 3d,e. For PCB1, 3 × 3 convolution effectively aggregates features and extends them to high-dimensional space. PConv efficiently process high-dimensional space features, and 1 × 1 convolution serves as a projection layer to project the upper output to low-dimensional space. For PCB2, when the step size is 2, two PConv are used to further expand the receptive field, and one PConv is used as a shortcut.

For the input image IM, the middle $X^{S1S2}$ is first obtained through the Stem block of the ESEB, followed by the feature $X^{S3S4}$ obtained through the proposed PCB block, and finally the semantic feature $X$ obtained through the CE block. The calculation process is as follows:

$$X^{S1S2}=Stem\left(IM\right)$$

(2)

$$X^{S3S4}=PCB{\left(X^{S1S2}\right)}_8$$

(3)

$$X=CE\left(X^{S3S4}\right)$$

(4)

where $Stem\left(·\right)$ represents the input feature passing through the Stem block, $PCB{\left(·\right)}_8$ represents the intermediate feature passing through the PCB block 8 times, and $CE\left(·\right)$ represents the intermediate feature passing through the CE block.

The details of the final backbone network are shown in

Table 1. Details of the backbone network.

Stage	LDEB					FOLPS	ESEB					FLOPS	Size
	opr	k	c	s	r		opr	k	c	s	r
Input													640 × 640
S1	Conv2d	3	64	2	1	5.1 G	Stem	3	32	4	1	165.7 M	320 × 320 320 × 320
	Conv2d	3	64	1	1
S2	Conv2d	3	128	2	1	3.0 G							160 × 160 160 × 160
	Conv2d	3	128	1	2
S3	Conv2d	3	256	2	1	2.4 G	PCB2	3	64	2	1	1.2 G	80 × 80 80 × 80
	Conv2d	3	256	1	2		PCB1	3	64	1	1
S4							PCB2	3	128	2	1	1.2 G	40 × 40 40 × 40
							PCB1	3	128	1	1
S5							PCB2	3	256	2	1	2.4 G	20 × 20 20 × 20 20 × 20
							PCB1	3	256	1	3
							CE	3	256	1	1

. The stage represents different stages of feature extraction, opr represents the operation, k represents the maximum kernel size in that operation, c represents the number of channels, s represents the stride, and r represents the number of repetitions.

3.2. Efficient Attention Aggregation

This section introduces the EAA module proposed in this paper. BiSeNet V2 [19] incorporates a BGA layer to fuse complementary information from two branches, where the BGA utilizes the contextual information from the high-level semantic branch to guide the feature response of the low-level detail branch. However, the simple guidance overlooks the difference in the importance of the two types of information, and the BGA employs DWConv with unexpanded channel capacity, which inevitably leads to a decrease in accuracy. To allow the model to focus more efficiently and adaptively on important semantic information and to fuse the output features of the two branches, we propose the EAA module. Building upon the BGA, the EAA module introduces TAF [20] and TCSA, while replacing the DWConv with PConv. The structure of the EAA module is illustrated in Figure 4.

3.2.1. Triplet Attention Fusion

The ESEB is characterized by a large channel capacity and low resolution. The large channel capacity contains rich semantic information, and establishing channel relationships is beneficial for focusing on important semantic features between channels. The low resolution contains limited spatial information, and establishing spatial relationships can easily lead to pixel confusion. In view of this, this paper adopted the TAF module [20] to establish channel relationships for ESEB. Compared with other state-of-the-art channel attention modules, TAF has fewer parameters and lower computational complexity, which is in line with the goal of constructing a lightweight network. Its structure is shown in Figure 5.

For a given input semantic feature $X\in R^{C\times H\times W}$, one global max pooling and two average pooling operations in space are performed to obtain global feature vectors $X_1$, $X_2$, $X_3$ of size $1\times 1\times \mathrm{C}$. The calculation process is as follows:

$$X_1=X_3=Avg\left(X\left(i,j\right)\right)$$

(5)

$$X_2=Max\left(X\left(i,j\right)\right)$$

(6)

where $Avg\left(·\right)$ represents the global average pooling operation and $Max\left(·\right)$ represents the global max pooling operation. $i\in \left\{1,2,\cdots ,H\right\}$, $j\in \left\{1,2,\cdots ,W\right\}$.

The global feature vectors $X_1$ and $X_2$ are transposed and then passed through a 1D convolution to obtain $X_{1-1}$ and $X_{2-1}$. Both paths share the same convolution kernel to enhance generalization while reducing the number of parameters. The calculation process is as follows:

$$X_{1-1}=Conv1D\left(trans\left(X_1\right)\right)$$

(7)

$$X_{2-1}=Conv1D\left(trans\left(X_2\right)\right)$$

(8)

where $trans\left(·\right)$ represents the transpose operation and $Conv1D\left(·\right)$ represents a 1D convolution operation with a kernel size of 3.

The global feature vector $X_3$ is used to capture the channel attention vector through a 2D convolution, and after reshaping the dimensions. The calculation process is as follows:

$$X_{3-1}=Conv\left(\delta \left(Conv\left(X3\right)\right)\right)$$

(9)

where $Conv\left(·\right)$ represents a 2D convolution operation with a kernel size of 1, and $\delta \left(·\right)$ represents the ReLU function.

After the fusion of $X_{3-1}$, $X_{1-1}$, and $X_{2-1}$, the channel attention vector V is obtained through activation and dimension reshaping operations. The calculation process is as follows:

$$V=trans\left(\sigma \left(X_{1-1}+X_{2-1}+trans\left(X_{3-1}\right)\right)\right)$$

(10)

where $\sigma \left(·\right)$ represents the Sigmoid function and $+$ represents the element by element addition.

The original feature map X is element-wise multiplied with the channel attention vector $V$ to obtain enhanced semantic features $X_V$. The calculation process is as follows:

$$X_v=X\times V$$

(11)

where × represents the element by element multiplication.

3.2.2. Triple Channel Spatial Attention

The LDEB has a higher resolution and larger channel capacity. The high resolution contains rich spatial information, while the large channel capacity contains rich semantic information. Establishing spatial and channel relationships is beneficial for capturing critical detail features. Drawing inspiration from the design architecture of CBAM [38], we proposed the TCSA module to establish spatial and channel relationships for LDEB. Its structure is shown in Figure 6.

Formally, for the input detail feature $X^{*}$, the TAF module is embedded to capture the enhanced detail feature $X_V^{*}$ of the feature map $X^{*}$. The calculation process is as follows:

$$X_V^{*}=TAF\left(X^{*}\right)$$

(12)

where $TAF\left(·\right)$ represents the TAF operation.

The enhanced detail features $X_V^{*}$ are embedded with spatial attention (SA), resulting in the spatial attention vector $S^{*}$. The calculation process is as follows:

$$S^{*}=\sigma \left(Con{v}_{7\times 7}\left[Avg\left(X_v\right);Max\left(X_v\right)\right]\right)$$

(13)

where $Con{v}_{7\times 7}$ represents the convolution operation with a kernel size of $7\times 7$.

The enhanced detail features $X_V^{*}$ are element-wise multiplied with the spatial attention vector $S^{*}$ to obtain the enhanced detail features $X_S$. The calculation process is as follows:

$$X_S=X_V^{*}\times S^{*}$$

(14)

3.2.3. Partial Convolution

PConv [37] replaces DWConv [35] and reduces redundant calculations and memory access, effectively reducing the parameters and computational complexity of the network. DWConv first performs feature extraction on individual input channels and then applies pointwise convolution to change the channel count. On the other hand, PConv only performs a convolution of partial channels, further incorporating pointwise convolution into PConv to form a T-shaped receptive field.

As shown in Figure 7, for the input $I\in R^{c\times h\times w}$, DWConv applies c convolution kernels of $W\in R^{k\times k}$ to compute the output $O\in R^{c\times h\times w}$, and it has a total of $h\times w\times k^2\times c$ FLOPs. Then, if it appends a pointwise convolution, the number of memory accesses increases to $h\times w\times c+k^2\times c$. On the other hand, PConv applies regular convolution for spatial feature extraction on only a subset of input channels while keeping the rest unchanged. Therefore, its FLOPs are only $h\times w\times k^2\times c_p{}^2$, when we take the partial ratio is $r=\frac{c_p}{c}=\frac{1}{4}$. As a result, the FLOPs of PConv are only half of that of DWConv. Additionally, PConv has lower memory access, with the appended pointwise convolution being only $h\times w\times c_p+k^2\times c_p$.

The guidance semantic feature $X_{SP}$ and interactive semantic feature $X_{SU}$ are obtained by enhancing the semantic feature $X_v$ through PConv and upsampling operations, respectively. The calculation process is as follows:

$$X_{SP}=PConv\left(X_V\right)$$

(15)

$$X_{SU}=Upsample\left(X_V\right)$$

(16)

where $PConv\left(·\right)$ represents the PConv operation and $Upsample\left(·\right)$ represents the upsampling operation.

The guidance detail feature $X_{DP}$ and interaction detail feature $X_{DA}$ are obtained by enhancing the detail feature $X_S$ through PConv and average pooling operations, respectively. The calculation process is as follows:

$$X_{DP}=PConv\left(X_S\right)$$

(17)

$$X_{DA}=Apooling\left(X_V\right)$$

(18)

where $Apooling\left(·\right)$ represents the general average pooling operation.

$X_{SU}$ guides $X_{DP}$ through element-wise multiplication to generate the final detailed feature $X_{gd}$, while $X_{DA}$ guides $X_{SP}$ through element-wise multiplication to generate the final semantic feature $X_{gs}$. The calculation process is as follows:

$${\mathrm{X}}_{\mathrm{gd}}={\mathrm{X}}_{\mathrm{SU}}\times {\mathrm{X}}_{\mathrm{DP}}$$

(19)

$${\mathrm{X}}_{\mathrm{gs}}={\mathrm{X}}_{\mathrm{DA}}\times {\mathrm{X}}_{\mathrm{SP}}$$

(20)

The final $X_{gd}$ and $X_{gs}$ are added element by element after the fused feature $F^{*}$ is obtained through $3\times 3$ convolution to eliminate aliasing effects. The calculation process is as follows:

$$F^{*}=Con{v}_{3\times 3}\left(X_{gd}+X_{gs}\right)$$

(21)

where $Con{v}_{3\times 3}\left(·\right)$ represents the $3\times 3$ convolution operation.

3.3. Feature Enhancement

In the processing by the EAA module, different branch feature maps may exhibit semantic differences, leading to inaccurate localization of text instances. To address this issue, we propose the FE Module, as illustrated in Figure 8. This module utilizes CA [21] to aggregate input features along the horizontal and vertical directions separately. It aims to capture position-sensitive information and channel long-range dependencies, compensating for the semantic discrepancy during the fusion process.

For a given fused features $F^{*}$, the projection features $Z^h$ and $Z^w$ along the horizontal and vertical axes, respectively, are obtained by performing average pooling using pooling kernels of size (H, 1) and (1, W). The calculation process is as follows:

$$Z^h=Poo{l}_h\left(F^{*}\right)$$

(22)

$$Z^w=Poo{l}_w\left(F^{*}\right)$$

(23)

where $Poo{l}_h\left(·\right)$ represents the average pooling operation along the vertical axis and $Poo{l}_w\left(·\right)$ represents the average pooling operation along the horizontal axis.

The semantic features $Z^h$ and $Z^w$ are concatenated along the spatial direction and then compressed in the channel dimension through a convolution operation to generate the intermediate feature f. The calculation process is as follows:

$$f=\delta \left(Con{v}_{1\times 1}\left(\left[Z^h,Z^w\right]\right)\right)$$

(24)

where $\left[·,·\right]$ represents the concatenation operation between two tensors and $Con{v}_{1\times 1}\left(·\right)$ represents the $1\times 1$ convolution operation.

The intermediate feature f is split along the spatial dimension into two separate tensors, $f^h$ and $f^w$. The $1\times 1$ convolution operation is applied to transform $f^h$ and $f^w$ into the same number of channels as the input $F^{*}$. The calculation process is as follows:

$$g^h=\sigma \left(Con{v}_{1\times 1}\left(f^h\right)\right)$$

(25)

$$g^w=\sigma \left(Con{v}_{1\times 1}\left(f^w\right)\right)$$

(26)

After the FE module, the final feature map $F$ is obtained. The calculation process is as follows:

$$F=\left(F^{*}\times g^h\times g^w \right)\oplus F^{*}$$

(27)

where $\oplus$ represents the element-wise addition.

3.4. Post-Processing

3.4.1. Differentiable Binarization

Traditional binarization is non-differentiable, which means it cannot directly be used for backpropagation and updating the network. PCBSNet uses a DB module, which allows for backpropagation and network updates. The expression for the module is as follows:

$$B_{i,j}=sigmoid\left(-k\left(P_{i,j}-T_{i,j}\right)\right)$$

(28)

where $B_{i,j}$ is the approximate binary mapping for pixel (i, j), $P_{i,j}$ is the probability map indicating the probability of pixel (i, j) being text, and $T_{i,j}$ is the adaptive threshold learned from the network for pixel (i, j). K is the amplification factor, set to 50, which is used to amplify the gradients during backpropagation.

3.4.2. Loss Function

The loss function consists of the probability map loss $L_s$, the binary map loss $L_b$, and the threshold map loss $L_t$, which are computed as follows:

$$L=L_s+\alpha \times L_t+\beta \times L_b$$

(29)

where $\alpha$ and $\beta$ are hyperparameters that determine the importance of each component of the loss. In this study, $\alpha$ was set to 1 and $\beta$ was set to 10.

The $L_s$ and $L_b$ are calculated using binary cross-entropy (BCE) loss, addressing the issue of imbalanced positive and negative samples through hard example mining.

The $L_t$ is defined as the L1 distance between the probability map label $y_i^{*}$ and the predicted value $x_i^{*}$, and it is computed as follows:

$$L_t={\displaystyle \sum }_{i\in R_d}\left|y_i^{*}-x_i^{*}\right|$$

(30)

4. Experiment

In this section, we validate the proposed PCBSNet through a series of experiments conducted on four publicly available datasets. These experiments aim to demonstrate its effectiveness and feasibility in terms of performance.

4.1. Dataset

The four publicly available datasets used in the experiments are described below.

SynthText [39]: It is a large-scale synthetic text detection dataset consisting of approximately 800,000 images. The dataset provides annotations for text instances in the form of words, characters, and text lines. In the experiments, this dataset was used for model pre-training.

ICDAR2015 [40]: This dataset focuses on multi-oriented text detection. It contains 1500 images, with 1000 images in the training set and 500 images in the test set. The text in the images is small in size and mainly composed of English and digits. The text regions are annotated with rectangular boxes represented by four vertices. The dataset has complex backgrounds and significant variations in text scales. It is commonly used for evaluating text detection algorithms. In the experiments, this dataset was used for model fine-tuning and algorithm performance validation.

MSRA-TD500 [41]: This dataset was also designed for multi-oriented text detection. It includes 500 images, with 300 images in the training set and 200 images in the test set. The dataset consists of English and Chinese text, and the text instances are annotated at the line level using four-coordinate rectangles.

CTW1500 [42]: This dataset focuses on curved text in multiple languages. It consists of 1000 training images and 500 test images. The annotations are in the form of polygons with 14 points.

These datasets are widely used in the field of text detection and provide diverse challenges to evaluate the performance and effectiveness of the proposed PCBSNet model.

4.2. Experimental Settings

The training environment where the model is located was an NVIDIA GeForce RTX 3090. BiSeNet V2 [19] was pretrained on SynthText [39] for the backbone network. It underwent pretraining for 100,000 iterations. During fine-tuning experiments, the batch size was set to 8, and the model was trained for 1200 epochs. The Adam optimizer [43] was used for parameter optimization with an initial learning rate of 0.007, weight decay of 0.0001, and momentum of 0.9. The learning rate was adjusted iteratively using the formula ${\left(1-\frac{iter}{max \_iter}\right)}^{power}$, where power was set to 0.9 and $max \_iter$ is the maximum number of iterations. The training samples were resized to 640 × 640 pixels.

Data augmentation was applied during the training phase, including random rotation, changes in brightness and saturation, random flipping, and random cropping.

To account for different experimental environments, the ablation experiments provided the locally reproduced experimental data for the BiSeNet V2 [19] baseline model, while the parameter experiments provided the locally reproduced experimental data for the DBNet [9] baseline model.

4.3. Evaluation Indicators

To evaluate the detection performance of the model, four commonly used text detection evaluation metrics were employed in this paper. FPS represents the number of frames processed per second, indicating real-time capability. A higher FPS indicates better real-time performance. Recall (R) measures the proportion of valid ground truth data correctly detected and is calculated using Formula (31). Precision (P) measures the proportion of correctly detected text out of all detected text instances and is calculated using Formula (32). The F-measure combines Precision and Recall and is calculated using Formula (33).

$$Recall=\frac{TP}{TP+FP}$$

(31)

$$Precision=\frac{TP}{TP+FN}$$

(32)

$$F-measure=\frac{2\times Recall\times Precision}{Recall+Precision}$$

(33)

In the formulas, TP represents the total number of text regions correctly predicted by the model, FP represents the total number of incorrectly predicted text regions, and FN represents the total number of text regions that were not predicted.

4.4. Ablation Experiments

To validate the effectiveness of the improvements in the ESEB, EAA module, and the overall model, we conducted extensive ablation experiments on the ICDAR2015 dataset. In the table, bold font indicates the best results for each metric. P, R, F, and FPS represent precision, recall, F-measure, and frames per second, respectively.

4.4.1. Efficient Semantic Extraction Branch Ablation Experiment

The ablation experiments for the ESEB involved the design of five comparative models on the ICDAR2015 dataset:

(1)

BVSE: Baseline model with the semantic extraction branch from BiSeNet V2 [19].

(2)

BVSE + W-C: ESEB with a wide range of channels.

(3)

BVSE + W-C + P: ESEB with wider channels and PConv.

(4)

BVSE + W-C + P + B-K: ESEB with wide channels, PConv, and large convolution kernels.

(5)

ESEB: ESEB with wide channels, PConv, large convolution kernels, and GELU activation function.

These models were designed to evaluate the effectiveness of the ESEB by gradually incorporating different enhancements to the semantic extraction branch.

The experimental results are shown in

Table 2. Experimental results of Efficient Semantic Extraction Branch ablation.

Model	ICDAR2015
	P%	R%	F%	FPS
BVSE	86.0	73.2	79.1	65.7
BVSE + W-C	86.8	73.8	79.8	58.9
BVSE + W-C + P	86.3	73.9	79.6	67.4
BVSE + W-C + P + B-K	87.4	74.4	80.4	66.2
ESEB	87.5	74.9	80.7	66.1

and demonstrate the impact of different enhancements on the ESEB. Compared to the baseline model, BVSE + W-C achieved a 0.7% improvement in the Fmeasure. However, BVSE + W-C + P, while achieving a 0.5% improvement compared to the baseline model, showed a 0.2% decrease compared to BVSE + W-C. BVSE + W-C + P + B-K, built upon BVSE + W-C + P, further improved the F-measure by 0.8%. When the ReLU activation function in BVSE + W-C + P was replaced with the GELU function, suitable for lightweight networks, ESEB demonstrated an improved convergence and achieved the best F-measure of 80.7%. These results indicate that the proposed ESEB effectively captures the semantic features of text in images, while accelerating the inference speed of model and enhancing its real-time performance. The decrease in the F-measure observed in the BVSE + W-C + P model can be attributed to the replacement of DWConv with PConv, which led to insufficient feature extraction saturation.

4.4.2. Efficient Attention Aggregation Module Ablation Experiment

The experimental design for the ablation study on the EAA module included seven comparative models:

(1)

BVBGA: Baseline model with the bidirectional guided aggregation module of BiSeNet V2 [19].

(2)

BVBGA + S-T: EAA module with semantic feature embedding and TAF.

(3)

BVBGA + D-TC: EAA module with detail feature embedding and TCSA.

(4)

BVBGA + S-TC: EAA module with semantic feature embedding and TCSA.

(5)

BVBGA + D-T: EAA module with detail feature embedding and TAF.

(6)

BVBGA + P: EAA module that replaces DWConv with PConv.

(7)

EAA: EAA module with semantic feature embedding and TAF, detail feature embedding and TCS, and replacement of DWConv with PConv.

These models were designed to evaluate the effectiveness of different enhancements in the EAA module.

The experimental results are shown in

Table 3. Experimental results of efficient attention aggregation module ablation.

Model	ICDAR2015
	P%	R%	F%	FPS
BVBGA	86.0	73.2	79.1	65.7
BVBGA + S-T	87.4	74.4	80.3	64.7
BVBGA + D-TC	87.1	74.0	80.0	64.2
BVBGA + S-TC	87.2	73.9	80.0	65.0
BVBGA + D-T	86.7	73.7	79.7	64.5
BVBGA + P	85.5	73.1	78.8	67.4
EAA	87.7	75.4	81.1	65.9

. BVBGA + S-T establishes channel relationships for semantic features, capturing key semantic information. Compared to the BVBGA baseline, it improved the F-measure by 1.2%. BVBGA + D-TC establishes channel and spatial relationships for detail features, capturing critical details. Compared to the BVBGA baseline, it improved the F-measure by 0.9%. BVBGA + S-TC adds spatial attention on top of the BVBGA + S-T, which causes confusion in semantic encoding. As a result, compared to the BVBGA + S-T, the F-measure decreased by 0.3%. BVBGA + D-T removes spatial attention from BVBGA + D-TC, leading to inadequate capture of detail features. Compared to BVBGA + D-TC, the F-measure decreased by 0.3%. BVBGA + P replaces DWConv with PConv in the baseline model, significantly improving the speed but slightly decreasing the F-measure by 0.2%. The EAA module incorporates the optimal improvements, including semantic feature embedding TAF, detail feature embedding with TCSA, and replacing DWConv with PConv. It achieved an F-measure of 81.1%. Based on the experimental results, the proposed EAA module effectively aggregates detail information and semantic information, improving the robustness of the model.

4.4.3. Overall Ablation Experiment

The experimental design for the overall ablation study included four comparative models:

(1)

BV: The baseline model based on BiSeNet V2 [19].

(2)

ESEB: A model where the original semantic branch was replaced with the ESEB.

(3)

ESEB + EAA: A model where the original semantic branch was replaced with the ESEB and the EAA module was added.

(4)

ESEB + EAA + FS: A model where the original semantic branch was replaced with the ESEB, and both the EAA module and the FE module were added.

The experimental results are shown in

Table 4. Overall ablation experimental results.

Model	ICDAR2015
	P%	R%	F%	FPS
BV	86.0	73.2	79.1	65.7
BV + ESEB	87.5	74.9	80.7	66.1
BV + ESEB + EAA	86.8	77.2	81.7	62.7
BV + ESEB + EAA + FS	88.0	78.4	82.9	59.1

. BV + ESEB replaces the original semantic branch with the ESEB, which improves the semantic extraction efficiency. It brought an 1.6% improvement in the F-measure to the baseline model, while slightly increasing the inference speed (by 0.04). BV + ESEB + EAA further replaces the original BGA module with the EAA module. By efficiently integrating the features from the dual branches using attention mechanisms and PConv adaptive guidance, the F-measure improved by an additional 1.0%, while the inference speed slightly decreased. BV + ESEB + EAA + FS enhances the fused features to obtain more accurate feature representations. As a result, the F-measure further improved by 1.2%, reaching 82.9% in total.

Figure 9 shows the loss curves of the different models in the overall ablation experiment.

4.4.4. Parameter Experiment

The Parameter experiment design used three comparative models:

(1)

Res + FPN: Network model using ResNet18 + FPN as adopted by DBNet [9].

(2)

BV + BGA: Network model using Bilateral Network + BGA as adopted by BiSeNet V2 [19].

(3)

Ours: Network model proposed in this paper.

The experimental results are shown in

Table 5. Parameter experiment results.

Model	Parameters	ICDAR2015
		P%	R%	F%	FPS
Res + FPN	11.36 M	91.0	72.1	80.4	44.4
BV + BGA	3.48 M	86.0	73.2	79.1	65.7
Ours	7.91 M	88.0	78.4	82.9	59.1

. Compared to the traditional feature processing mode of ResNet18 + FPN, the dual-branch structure of BiSeNet V2 [19] had a parameter size of only 3.48 M, resulting in a significant improvement in inference speed. However, there was a slight decrease in the three evaluation metrics. The proposed model achieved the optimal values in all three evaluation metrics and had an increase of 4.43 M parameters compared to BiSeNet V2 [19], but it was still much lower than ResNet + FPN.

Figure 10 displays the partial detection results and corresponding binary images of the proposed model on the three datasets.

4.5. Comparative Experiment

To further validate the effectiveness of the proposed detection method, comparisons were made with recent mainstream lightweight methods on the ICDAR2015 [40] dataset, MSRA-TD500 [41] dataset, and CTW1500 [42] dataset. The segmentation-based detection methods all use ResNet18 as the backbone network without deformable convolutions. The comparative results on the three datasets are shown in Figure 11, Figure 12 and Figure 13. The experiments were conducted using the SynthText [39] dataset for pre-training, followed by 1200 iterations of fine-tuning training. The detailed experimental results are presented in Table 6, Table 7 and Table 8.

According to the experimental results shown in Table 6, our proposed PCBSNet benefits from lightweight strategies and significantly improves the inference speed, achieving a state-of-the-art (SOTA) FPS 59.1 frames per second when faced with datasets like ICDAR2015 that contain small and low-resolution text instances. However, due to the abundance of small text instances in this dataset, our feature extraction capability was slightly inadequate, leading to the possibility of missing detections of small text instances and resulting in a third-best Precision of 88.0%. In comparison to the methods with the best Precision and Recall, DBNet++ [44] and PAN [22], the former has a complex model that overfits the training data, resulting in high Precision and low Recall, while the latter has a simple model that fails to capture complex data patterns and relationships, leading to more false positives and a trade-off between Recall and Precision. As a result, neither of these methods achieved SOTA results in terms of the F-measure. In contrast, our PCBSNet, with its effective fusion mechanism and feature enhancement, achieved a SOTA F-measure of 82.9%.

According to the experimental results shown in Table 7, our proposed PCBSNet achieves SOTA Precision, F-measure, and FPS when dealing with smaller datasets like MSRA-TD500, reaching 90.0%, 82.8%, and 94.3 frames per second, respectively. The MSRA-TD500 dataset contains larger text instances with high data quality and a balanced distribution of positive and negative samples, eliminating the need for deep convolutional networks like ResNet used by DBNet++ [44] or VGG used by Corner [45]. Our lightweight dual-branch approach effectively extracts semantic information and detailed information and combines them through attention fusion, resulting in accurate segmentation outcomes.

According to the experimental results shown in Table 8, PCBSNet falls short of achieving SOTA results when faced with curved text datasets like CTW1500, achieving only the second-best FPS 75.5 frames per second and the third-best Precision of 83.8%. The reason for obtaining high Precision but low Recall is that our lightweight PCBSNet lacks the feature representation capability required to adapt to the complexity and diversity of curved text, which typically necessitates more complex models. For example, LOMO [46] achieved the best Precision by employing ResNet for comprehensive feature extraction and FPN for multi-scale fusion, but it falls short in terms of Recall, while TextSnake [47] achieved the best Recall by treating curved text as a shape composed of continuous line segments for accurate segmentation but sacrifices precision by incorrectly segmenting a large portion of non-curved text.

In conclusion, our proposed PCBSNet, which incorporates a lightweight backbone for real-time semantic segmentation, demonstrates outstanding inference speed compared to other methods when handling text in various natural scenes, meeting the requirements of real-time applications. On datasets like ICDAR2015 [40] and MSRA-TD500 [41], PCBSNet achieves accurate segmentation results for text of various orientations, scales, and shapes, and it also performs competitively with small and low-contrast text. However, challenges still exist when dealing with curved text in the CTW1500 [42] dataset, as our model, designed to accommodate quadrilateral text annotations, fails to accurately capture the boundary features of curved text, resulting in false detections. Through experimental analysis, PCBSNet demonstrates good scalability when faced with horizontal text in flat advertisements, book pages, newspapers, as well as vertical text in signs, banners, and store names, and it can handle texts in multiple languages such as English and Chinese. Furthermore, its superior inference speed makes it well-suited for various real.

Based on the aforementioned experimental results, our proposed PCBSNet, with its lightweight backbone for real-time semantic segmentation, demonstrates outstanding inference speed compared to other methods, fulfilling the requirement for real-time applications. On the ICDAR2015 [40] and MSRA-TD500 [41] datasets, PCBSNet achieved accurate segmentation results for text of various orientations, scales, and arbitrary shapes, particularly for high-quality text. It also performed competitively on small text and low-contrast text. However, challenges still persist when dealing with curved text, as observed in the CTW1500 [42] dataset. Due to the adaptation to quadrilateral text annotations, the model struggles to accurately capture the boundary features of curved text, resulting in false detections. Through experimental analysis, PCBSNet exhibited good scalability when faced with horizontal text such as advertisements, book pages, and newspapers, as well as vertical text such as signs, banners, and store names. It also showed versatility in detecting text in multiple languages, including English and Chinese. Moreover, thanks to its excellent inference speed, PCBSNet was capable of handling text detection tasks in various real-time application scenarios where a quick response is crucial.

Table 6. ICDAR2015 comparative experimental results.

Model	ICDAR2015
	P%	R%	F%	FPS	Params
TextSnake [47]	84.9	80.4	82.6	1.1	19.1 M
TextField [48]	84.3	80.5	82.4	5.5	17 M+
PSENet [23]	88.2	74.2	80.6	1.6	28.6 M
PAN [22]	79.3	84.0	81.8	20.3	14 M+
DBNet [9]	86.8	77.7	82.0	44.4	11.4 M
PAN++ [49]	85.9	79.9	82.8	28.2	-
DBNet++ [44]	90.1	75.1	82.8	40.2	12.4 M
Ours	88.0	78.4	82.9	59.1	7.91 M

Table 6. ICDAR2015 comparative experimental results.

Model	ICDAR2015
	P%	R%	F%	FPS	Params
TextSnake [47]	84.9	80.4	82.6	1.1	19.1 M
TextField [48]	84.3	80.5	82.4	5.5	17 M+
PSENet [23]	88.2	74.2	80.6	1.6	28.6 M
PAN [22]	79.3	84.0	81.8	20.3	14 M+
DBNet [9]	86.8	77.7	82.0	44.4	11.4 M
PAN++ [49]	85.9	79.9	82.8	28.2	-
DBNet++ [44]	90.1	75.1	82.8	40.2	12.4 M
Ours	88.0	78.4	82.9	59.1	7.91 M

Table 7. MSRA-TD500 comparative experimental results.

Model	MSRA-TD500
	P%	R%	F%	FPS	Params
TextField [48]	87.4	75.9	81.3	5.2	32.7 M
TextSnake [47]	83.2	73.9	78.3	1.1	19.1 M
RRD [50]	87.0	73.0	79.0	10.0	-
Corner [45]	87.6	76.2	81.5	5.7	45.2 M
PAN [22]	80.7	77.3	78.9	30.2	14 M+
DBNet [9]	85.7	73.2	79.0	82.0	11.4 M
DBNet++ [44]	89.7	76.5	82.6	80.0	12.4 M
Ours	90.0	76.7	82.8	94.3	7.91 M

Table 7. MSRA-TD500 comparative experimental results.

Model	MSRA-TD500
	P%	R%	F%	FPS	Params
TextField [48]	87.4	75.9	81.3	5.2	32.7 M
TextSnake [47]	83.2	73.9	78.3	1.1	19.1 M
RRD [50]	87.0	73.0	79.0	10.0	-
Corner [45]	87.6	76.2	81.5	5.7	45.2 M
PAN [22]	80.7	77.3	78.9	30.2	14 M+
DBNet [9]	85.7	73.2	79.0	82.0	11.4 M
DBNet++ [44]	89.7	76.5	82.6	80.0	12.4 M
Ours	90.0	76.7	82.8	94.3	7.91 M

Table 8. CTW1500 comparative experimental results.

Model	CTW1500
	P%	R%	F%	FPS	Params
TextSnake [47]	67.9	85.3	75.6	1.1	19.1 M
SAE [51]	82.7	77.8	80.1	3	27.8 M
MASK TTD [52]	79.7	79.0	79.4	-	28 M+
PAN [23]	82.7	77.4	79.9	84.2	14 M+
LOMO [46]	89.2	69.9	78.4	-	-
DBNet [9]	82.4	76.6	79.4	59	11.4 M
DBNet++ [44]	84.8	76.3	80.3	59	12.4 M
Ours	83.8	74.5	78.9	75.5	7.91 M

Table 8. CTW1500 comparative experimental results.

Model	CTW1500
	P%	R%	F%	FPS	Params
TextSnake [47]	67.9	85.3	75.6	1.1	19.1 M
SAE [51]	82.7	77.8	80.1	3	27.8 M
MASK TTD [52]	79.7	79.0	79.4	-	28 M+
PAN [23]	82.7	77.4	79.9	84.2	14 M+
LOMO [46]	89.2	69.9	78.4	-	-
DBNet [9]	82.4	76.6	79.4	59	11.4 M
DBNet++ [44]	84.8	76.3	80.3	59	12.4 M
Ours	83.8	74.5	78.9	75.5	7.91 M

5. Conclusions

In order to address the real-time challenges of text detection in natural scenes, a high-efficiency and accurate real-time scene text detection model based on a pure convolutional bilateral segmentation network was proposed. It significantly improves the feature extraction efficiency by constructing a lightweight dual-branch architecture, and then efficiently and adaptively fuses the dual-branch output features through an efficient attention aggregation module. Additionally, a coordinate attention mechanism was introduced after feature fusion to construct a feature enhancement module, alleviating the distortion phenomenon in the fusion process. A series of experiments conducted on the ICDAR2015, MSRA-TD500, and CTW1500 datasets demonstrated that the proposed PCBSNet achieves a good balance between accuracy and real-time performance. However, like most methods, there is still room for improvement in the performance of PCBSNet when dealing with curved text.

In the future, we will improve the fast downsampling strategy used in the underlying detail branches to find more convenient ways to capture low-level information and further enhance the real-time performance of the model. We will also explore whether expanding the number of channels in the PCB blocks by six times is the optimal structure, and conduct experiments to find the best structure for improving scene text detection performance.