Cascade Network with Deformable Composite Backbone for Formula Detection in Scanned Document Images

Abstract

This paper presents a novel architecture for detecting mathematical formulas in document images, which is an important step for reliable information extraction in several domains. Recently, Cascade Mask R-CNN networks have been introduced to solve object detection in computer vision. In this paper, we suggest a couple of modifications to the existing Cascade Mask R-CNN architecture: First, the proposed network uses deformable convolutions instead of conventional convolutions in the backbone network to spot areas of interest better. Second, it uses a dual backbone of ResNeXt-101, having composite connections at the parallel stages. Finally, our proposed network is end-to-end trainable. We evaluate the proposed approach on the ICDAR-2017 POD and Marmot datasets. The proposed approach demonstrates state-of-the-art performance on ICDAR-2017 POD at a higher IoU threshold with an f1-score of 0.917, reducing the relative error by 7.8%. Moreover, we accomplished correct detection accuracy of 81.3% on embedded formulas on the Marmot dataset, which results in a relative error reduction of 30%.

1. Introduction

Information extraction from document images is a primary need in various domains such as banking, archiving, or academia and industry in general. Research in document analysis has been trying to develop precise information extraction systems for several years [1,2,3,4]. Although state-of-the-art optical character recognition (OCR) systems [5,6] recognize regular text with high accuracy, they are vulnerable to recognize information from page objects (tables, figures, mathematical formulas) in document images [7,8]. Figure 1 illustrates the problem in which an open-source OCR, Tesseract [4] (we use the LSTM-based version 4.1.1 available at https://github.com/tesseract-ocr/tesseract accessed on 5 July 2021), is applied to extract the content from a document image. Besides recognizing the textual content, the OCR fails to extract the information from mathematical formulas. This shows that formula detection is a crucial preliminary step for information extraction in such document images.

Mathematical formulas are an integral part of documents because they allow us to represent complex information concisely by exploiting mathematics capabilities. Formulas present in the documents are categorized into isolated formulas (mentioned in a separate line) and embedded formulas (inline mathematical symbols). Figure 2 exhibits the problem of detecting isolated and embedded formulas in document images.

The task of detecting both isolated and embedded formulas in document images is a difficult problem because of the underlying low inter-class and high intra-class variance [10]. The hurdles involved in detecting isolated and embedded formulas are exhibited in Figure 2. The isolated formulas present in a document image can easily be misclassified with other page objects due to low inter-class variance with tables, algorithms, and figures. The embedded formulas contain mathematical functions ($log,exp,tan$), operators ($\times ,+,\sigma ,\%$), and variables ($i,j,k$). These inline expressions are prone to be misinterpreted with the regular text in a document image [11].

Previous works employed hand-crafted features to detect formulas in documents [2,12,13]. Although these systems extract mathematical formulas, they fail to obtain effective results on generic datasets. Later, statistical learning, mainly machine learning-based methods, advanced the performance of formula identification systems [14,15,16]. The recent success of deep learning-based methods on computer vision within the last decade also had an impact on the task of formula detection in scanned document images. Several deep learning-based formula detection approaches [17,18,19,20] have been presented in the past two years. They are mainly equipped with object detection algorithms such as Faster R-CNN [21], YOLO [22], SSD [23], and FPNs [24].

In recent work, Agarwal et al. [25] presented a method equipped with Cascade Mask R-CNN [26] to tackle the problem of table detection in document images. However, the capabilities of Cascade Mask R-CNN have not been investigated yet in the domain of mathematical formula detection in document images.

This paper presents an end-to-end data-driven approach to detect both isolated and embedded formulas in document images. The main contributions of this paper are as follows:

• We present an end-to-end trainable framework that operates on a Cascade Mask R-CNN equipped with a deformable composite backbone to detect both isolated and embedded formulas in document images.
• Unlike prior work, our formula detection pipeline operates on a lightweight dilation method as a pre-processing step.
• We accomplish state-of-the-art results in detecting isolated formulas on a higher IoU threshold in the ICDAR-2017 POD dataset [27]. Furthermore, on the Marmot dataset [9], we surpass previous state-of-the-art results on embedded formulas with a huge margin and achieve identical results with prior state-of-the-art on isolated formulas.

2. Related Work

Research progress in the field of document image analysis directly relates to advances in the computer vision research community. The task of formula detection in documents is a well-studied problem [28]. Noticeable progress has been achieved in this domain by implementing custom-heuristics to deep learning-based approaches. Earlier, rule-based approaches developed character-based heuristics to identify formulas in documents [29,30,31,32]. These techniques look for special characters (e.g., “>”, “×”, “=”) that mainly exist in mathematical formulas.

Kacem et al. [12] introduced a model based on fuzzy logic to detect mathematical symbols. The approach predicts the formula region by exploiting the features of mathematical symbols. Inoue et al. [2] first employed a conventional OCR method to extract characters. The method treated all the remaining characters as mathematical symbols that OCR was unable to parse.

Specific OCR systems have been presented that recognize mathematical symbols based on their positions and sizes [2]. Baker et al. [13] segregated the lines containing formulas to the regular textual lines in order to detect isolated formulas in PDF documents.

Decision trees have been equipped to detect isolated formulas by classifying formula lines with the plain text lines [33]. Chang et al. [15] proposed a similar method based on the projection of the features that only works for isolated formulas in documents.

Later, machine learning-based algorithms were proposed to alleviate the performance of formula detection systems in documents [14,34]. Liu et al. [16] leveraged the combination of Conditional Random Field (CRF) and Support Vector Machine (SVM) to classify sparse lines in documents. Subsequently, the method distinguished formulas from other graphical page objects such as figures and tables by applying custom heuristics.

Succeedingly, researchers have investigated the capabilities of Deep Neural Networks (DNNs) for the problem of formula identification in document images [27,35]. To the best of our knowledge, He et al. [36] exploited Convolutional Neural Networks (CNNs) with spatial context to detect mathematical symbols in document images. Later, Gao et al. [37] presented a deep learning-based formula detection system in PDF documents.

NLPR-PAL [27] produced the best results in the competition of POD at ICDAR-2017. They proposed a blend of connected components, SVM, and Faster R-CNN [21] to detect figures, formulas, and tables in document images.

Yi et al. [38] published another CNN-based approach that detects graphical page objects such as tables, figures, and formulas in document images. The authors employed the dynamic programming technique instead of Non-Maximum Suppression (NMS) to refine the final candidate proposals. Semantic segmentation-based architecture such as U-Net [39] has also been utilized to detect mathematical expressions in scientific document images [17].

Recently, Phong et al. [18] published a method equipped with YOLO [40] to detect mathematical formulas in document images. In another approach [19], SSD [23] was exploited to detect mathematical expressions in PDF documents.

Another graphical page object detection system was published by Li et al. [41]. The authors combined deep structure prediction with a traditional approach to detecting page objects, including formulas in document images. Younas et al. [20] introduced a system called Fi-Fo that detects figures and formulas in document images. The authors empirically established that deformable convolutions [42] with Feature Pyramid Networks (FPN) [24] are a better fit as compared to other object detection algorithms. The proposed approach heavily relied on the image transformation pre-processing techniques to produce state-of-the-art results.

3. Method

The presented approach is comprised of Cascade Mask R-CNN [43] equipped with a recently published composite backbone having deformable convolutions replaced with traditional convolution filters. Figure 3 illustrates the complete pipeline of our proposed framework. In this section, we dive deeper into each component of our proposed method.

3.1. Cascade Mask R-CNN

We treat the problem of formula detection in document images as an object detection problem on natural images. Recently, Cai and Vasconcelos [26] introduced Cascade R-CNN [26] that extends the concept of the idea of Faster R-CNN [21] by adding multi staging technique. In our approach, we incorporate the instance segmentation branch as proposed in the original Mask R-CNN [43].

As explained in Figure 3, the input image is passed through the composite ResNeXt-101 backbone, which is explained in Section 3.2. The backbone extracts the spatial features and generates feature maps. The Region Proposal Network (RPN) head estimates the possible candidate regions where formulas can be present. The first bounding box component receives the features from the RPN and creates predictions. Each of the three bounding box modules performs classification and regression. The classification score and bounding box coordinates predicted by each bounding box head, $BH1$, $BH2$, and $BH3$, are denoted with $(C1,B1)$, $(C2,B2)$, and $(C3,B3)$, respectively. The output of one bounding box head becomes the training input for the next head. This cascaded regression and classification method optimizes the process of differentiating false positive samples with true positives even at higher IoU thresholds. After computing the refined bounding boxes and classification scores from $BH3$, the segmentation head predicts the mask that contributes to the loss function to optimize the training further.

3.2. Composite Backbone

We employ a robust and novel dual backbone architecture to extract the possible spatial features to detect formulas in document images. The performance of any object detection algorithm depends on the quality of the feature map it receives from the feature extraction network [44]. In this paper, we implement a dual backbone-based network [45] in which the first backbone is the assistance backbone, and the other is known as the lead backbone. Both of the backbones are compositely connected to each other so that the assistant backbone’s output features are treated as input features for the lead backbone. Figure 4 illustrates the architecture of our dual composite backbone.

For the conventional convolutional network with single backbone, the output of ($l-1$)-th stage is propagated as input to the l-th stage, which is given by

$$x{}^l=F^l\left(x^{l-1}\right),l\ge 2$$

(1)

where $F^l$ represents the non-linear function on l-th level. Contrary to this, our backbone network receives input from prior levels and parallel level of the assistant backbone. Therefore, the input of a lead backbone $bl$ at stage l is the product of output of lead backbone at ($l-1$)th stage and parallel $l-th$ stage of assistant backbone $ba$. Mathematically, it is explained in [45] as

$$x{}_{bl}^l=F{}_{bl}^l(\left(x_k^{l-1}\right)+g\left(x_{ba}^l\right)),l\ge 2$$

(2)

where g defines the composite connection between the lead and assistant backbone, and these composite connections enable the lead backbone to extract essential spatial features.

Table 1. Architectural details of the employed dual composite ResNeXt-101 backbone network. DCN represents the incorporation of deformable convolution.

Stage	Output	DCN	ResNeXt-101 (32 × 4d)
conv1	112 × 112	✗	7 × 7, 64, stride 2
conv2	56 × 56	-	3 × 3 max pooling, stride 2
		✗	1 × 1, 128 3 × 3, 128, C = 32 × 3 1 × 1, 128
conv3	28 × 28	✓	1 × 1, 256 3 × 3, 256, C = 32 × 4 1 × 1, 512
conv4	14 × 14	✓	1 × 1, 512 3 × 3, 512, C = 32 × 23 1 × 1, 1024
conv5	7 × 7	✓	1 × 1, 1024 3 × 3, 1024, C = 32 × 3 1 × 1, 2048
	1 × 1	✗	global average pool 1000-d fc, softmax

outlines the architectural details of the employed dual ResNeXt-101 backbone network. As explained in Figure 3, we propagate the output of the final lead backbone to the region proposal network of our Cascade Mask R-CNN.

3.3. Deformable Convolution

We incorporate deformable convolution filters [42] instead of the conventional convolutions that exist in the ResNeXt-101 architecture [46]. The convolutional neural networks extract the important spatial features that are essential to perform the required task. Based on the hierarchy, convolutional layers discover different features [47]. Convolutional layers present at the bottom search for crude features such as sharp edges or the gradients, whereas the layers at higher levels look for the abstract components such as complete object [48]. The conventional convolution operation has the same effective receptive field for all the neurons. The 2D convolution is comprised of two parts: (1) the first step samples the input feature map through a grid R, and (2) aggregation of sample values is multiplied by the weight $\mathbf{w}$. For conventional convolution, the output of feature map y for each position $p_o$ is elaborated in [42] as follows:

$$\mathbf{y}\left({\mathbf{p}}_0\right)=\sum _{{\mathbf{p}}_n\in \mathcal{R}}\mathbf{w}\left({\mathbf{p}}_n\right)\times \mathbf{x}\left({\mathbf{p}}_0+{\mathbf{p}}_n\right)$$

(3)

where x represents the input feature map, and $p_n$ iterates over the locations in a grid R that can be defined as $R=(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1)$ for a 3 × 3 convolutional layer. The effective receptive field of such a filter is restricted to these nine positions.

In the case of deformable convolution, an additional offset represented as $\Delta \left(p_n\right)$ is added, which deforms the filter’s receptive field by augmenting the predefined offsets. Hence, Equation (3), as explained in [42], is transformed into

$$\mathbf{y}\left({\mathbf{p}}_0\right)=\sum _{{\mathbf{p}}_n\in \mathcal{R}}\mathbf{w}\left({\mathbf{p}}_n\right)\times \mathbf{x}\left({\mathbf{p}}_0+{\mathbf{p}}_n+\Delta {\mathbf{p}}_n\right)$$

(4)

This modification makes the sampling process irregular with an offset value of $p_n+\Delta \left(p_n\right)$. As these offsets are differentiable and fractional, bilinear interpolation is used to implement them. Considering $p=p_0+p_n+\Delta \left(p_n\right)$, the bilinear interpolation is implemented as follows:

$$\mathbf{x}\left(\mathbf{p}\right)=\sum _{\mathbf{q}}\mathbf{G}\left(\mathbf{q},\mathbf{p}\right)\times \mathbf{x}\left(\mathbf{q}\right)$$

(5)

where q iterates over all the possible places on the input feature map x, and the symbol G represents the bilinear interpolation kernel. It is vital to mention that G is a two-dimensional kernel that can be further divided into two one-dimensional kernels. It is mathematically explained as

$$\mathbf{G}\left(\mathbf{q},\mathbf{p}\right)=\mathbf{g}\left({\mathbf{q}}_{\mathbf{x}},{\mathbf{p}}_{\mathbf{x}}\right)\times \mathbf{g}\left({\mathbf{q}}_{\mathbf{y}},{\mathbf{p}}_{\mathbf{y}}\right)$$

(6)

where g is explained as $g(a,b)=max(0,1-|a-b\left|\right)$. It is important to note that Equation (5) is more efficient since $G(q,p)$ is zero for most of the $\mathbf{q}s$. We refer our readers to [42,49] for a detailed explanation of deformable convolutions. Figure 5 depicts the architecture of deformable convolution. In order to convert our composite backbone network into its deformable counterparts, we replace conventional convolutional operation with deformable convolution in the higher-level layers that are from stage $C_3$ to stage $C_5$. Table 1 highlights the presence of deformable convolution in the backbone network.

3.4. Image Transformation and Prepossessing

Document images mainly consist of textual regions. There exists a variable amount of gap between textual components. This gap not only separates the textual components but also provides a higher level of semantic representation. We can think of formula detection as a semantic labeling task where a textual unit is labeled as a formula or other text depending upon its contents. In order to group closely related regions, we apply dilation transformation on the images. The dilation transformation converts the input images to semantically enriched representation. It is crucial to understand that this grouping cannot replace the actual image content. Therefore, we concatenate the prepossessed images with the original images. This concatenation increases the number of input channels. The deep neural network processes this combination.

Dilation Transformation

The dilation transformation is used to thicken the black regions in the input image. Since this transformation works on binary images, the input images are binarized first. The black pixel represents the characters, and the white pixels describe the background in the binarized images. Therefore, this transformation thickens the characters. Figure 6 depicts the output of dilation transformation on one of the sample images. We use a structuring element of $2\times 2$. We tried different sizes of the structuring elements. However, $2\times 2$ produces the optimal results.

3.5. Datasets

We employed the well-known publicly available formula detection datasets to conduct our experiments. This section elaborates upon these datasets, and their summary is presented in

Table 2. Summary of the main statistics of the employed datasets.

Datasets	ICDAR-17		Marmot
	Train	Test	Train	Test
Number of Images	1600	817	330	70
Number of Isolated Formulas	3534	1929	1322	253
Number of Embedded Formulas	-	-	6951	956

.

3.5.1. ICDAR-17

ICDAR-17 is the result of a recent competition in graphical page object detection (POD) [27] in document images at ICDAR in 2017. There are 2417 document images in the dataset having annotations for figures, formulas, and tables in document images. In addition, the dataset contains a variety of isolated formulas present on the single and multi-column document images. For the experiments, we have used 1600 images for training and 817 images for testing purposes. Recently, Younas et al. [20] published the corrected version of this dataset which leads to more formulas in the dataset. Therefore, we have employed the revised version of the dataset in our experiments for direct comparison with state-of-the-art results.

3.5.2. Marmot

Marmot [50] is fairly a smaller dataset consisting of 400 scanned document images. However, the dataset contains annotations for isolated and embedded mathematical equations. There are 1575 isolated formulas varying from 4 to 20 formulas per document image, whereas there are 7907 embedded formulas with an average of almost 20 embedded formulas per document image.

4. Experimental Results

4.1. Model Configuration

We implement the proposed method in Pytorch by leveraging the MMdetection object detection pipeline [51]. Our composite backbone ResNeXt-101 [46] is pre-trained on the MS-COCO dataset [52]. The pre-trained feature extraction network facilitates our object detection algorithm to adapt from the domain of natural scenes to documents. We scaled the input document images to 1200 × 800 but maintained the original aspect ratio. The training starts with a learning rate of 0.0025, which is reduced after every eighth epoch. We train the network for a total of 20 epochs for both of the datasets. The IoU threshold values for cascaded bounding boxes are set to [0.5, 0.6, 0.7]. We employed three different anchor ratios of [0.5, 1.0, 2.0] with only one anchor scale of [8] since FPN [24] itself performs the multi-scale detection owing to its top-down architecture. We operated with a batch size of one to train our network. The models for both of the datasets are trained on an NVIDIA GeForce RTX 101 Ti GPU with 12 GB memory.

4.2. Evaluation Metrics

For ICDAR-2017 POD, we work with the same evaluation criteria as elaborated in the ICDAR-2017 POD competition [27]. For the Marmot dataset, we follow the identical criteria of computing detection accuracy as explained in [11] to have direct comparisons. We report results by employing the following metrics.

4.2.1. Precision

Precision [53] defines the ratio of positive samples over all the predicted samples. Mathematically, it is given by

$$\mathrm{Precision}=\frac{\mathrm{True}\mathrm{Positives}}{\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Positives}}$$

(7)

4.2.2. Recall

Recall [53] calculates the ratio of positive samples in predictions over all the positive samples present in the ground truth. It is explained as follows:

$$\mathrm{Recall}=\frac{\mathrm{True}\mathrm{Positives}}{\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Negatives}}$$

(8)

4.2.3. F1-Score

The metrics f1-score [53] is the measure that is computed by taking the harmonic mean of precision and recall. The formula for f1-score is

$$\mathrm{F}1\text{-}\mathrm{Score}=\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(9)

4.2.4. Mean Average Precision (mAP)

The mean average precision, also referred to as mAP score, is calculated by averaging maximum precision over various recall thresholds. Mathematically, it is explained in [52] as follows:

$$\mathrm{mAP}=\frac{1}{N}\sum _{r=1}^{N}A{P}_r$$

(10)

where $A{P}_r$ is the average precision on a recall level r.

4.2.5. Intersection over Union (IOU)

The metrics Intersection over union [54] estimates the amount of predicted region intersecting with the ground truth region. It is explained as follows:

$$\mathrm{IoU}(\mathrm{A},\mathrm{B})=\frac{\mathrm{Area}\mathrm{of}\mathrm{Overlap}\mathrm{region}}{\mathrm{Area}\mathrm{of}\mathrm{Union}\mathrm{region}}=\frac{|A\cap B|}{|A\cup B|}$$

(11)

4.2.6. Detection Accuracy

We report results on the Marmot dataset using the metrics of detection accuracy. As explained in [11], we classify the prediction into correct and partially correct based on the IoU value:

• Correct: the predicted bounding box is considered correct when the IoU score between the predicted formula region and the ground truth is equal to or greater than 0.5.
• Partial: when the IoU score between the inferred and the ground truth formula region is in the interval (0; 0.5), the detection is categorized as partial.

4.3. Result and Discussion

We report the results on the datasets of ICDAR-2017 POD [27] and Marmot [9] to demonstrate the effectiveness of the proposed method. This section analyzes the qualitative and quantitative performance of our approach by highlighting the strengths and weaknesses. Furthermore, it compares the presented results with prior state-of-the-art methods.

4.3.1. ICDAR-17

We follow the evaluation protocol as elaborated in ICDAR-2017 POD [27]. We first calculate the number of true positives, false positives, and false negatives from the complete test set. We then compute the precision, recall, and F1-Score as calculated in the prior methods [20,41]. Moreover, we also report the mAP score by evaluating the performance of our method on the test set. Following the criteria of the competition, we present results on the IoU threshold of 0.6 and 0.8. It is essential to emphasize that we have employed the recently published corrected version of the dataset [20]. Therefore, only the methods that have reported results on the corrected version of the dataset are directly comparable with our approach.

Table 3. Quantitative analysis of the presented work with existing state-of-the-art methods on the ICDAR-2017 POD dataset. † represents the results that are not directly comparable with our method because they are not evaluated on the revised version of the dataset.

ICDAR-2017 POD
Method	IoU = 0.6				IoU = 0.8
	Precision	Recall	F1-Score	AP	Precision	Recall	F1-Score	AP
NLPR-PAL [27] †	0.901	0.929	0.915	0.839	0.888	0.916	0.902	0.816
Li et al. [41] †	0.935	0.331	0.489	0.312	0.877	0.310	0.459	0.274
Fi-Fo Detector Non Deformable [20]	0.910	0.927	0.918	0.953	0.860	0.877	0.868	0.928
Fi-Fo Detector Deformable [20]	0.957	0.952	0.954	0.949	0.913	0.908	0.910	0.898
Ours (Without Pre-Processing)	0.950	0.948	0.949	0.97	0.914	0.912	0.913	0.949
Ours (Complete Method)	0.954	0.952	0.953	0.97.5	0.918	0.916	0.917	0.954

presents the results that are achieved by our proposed end-to-end method with and without incorporating the pre-processing technique. After setting an IoU threshold of 0.6, we achieve a precision of 0.95, recall of 0.948, f1-score of 0.949, and mAP of 0.97 without the inclusion of the pre-processing method. The results further improve with an average of almost 0.04 after employing the proposed pre-processing. Upon increasing the IoU threshold value to 0.8, our network reaches a precision of 0.914, recall of 0.912, f1-score of 0.913, and mAP score of 0.949 in the absence of pre-processing method, and the presence of pre-processing advances the results with an average difference of 0.04. For the completeness of the paper, we compute the f1-score of the proposed method on different IoU thresholds ranging from 0.5 to 1.0. Figure 7 illustrates the performance of our approach in terms of f1-score.

Figure 8 and Figure 9 depict the qualitative performance of our proposed system. Out of 1929 isolate formulas present in the test set, our cascade formula detection network correctly predicted the region for 1836 formulas at an IoU threshold of 0.6. Moreover, it is vital to mention that even at a higher IoU threshold of 0.8, the system identified correct boundaries for 1767 formulas present in the test set. We also observe some rare cases of false positive and false negative samples, which are exhibited in Figure 9.

Comparison with State-of-the-Art Methods

By looking at Table 3, it is evident that our hybrid method of cascade network leveraging deformable composite backbone with lightweight pre-processing has outperformed the prior state-of-the-art method [20] on a higher IoU threshold of 0.8 with an average f1-score of 0.917, thus reducing the relative error by 7.8%. Furthermore, we achieve an almost identical f1-score at an IoU threshold of 0.6. It is essential to emphasize that the previous state-of-the-art work [20] depends on the heavy pre-processing pipeline consisting of distance transform and connected components analysis (CCA) applied on grayscale images. However, our generic data-driven method operates on the lightweight dilation technique to produce better results.

4.3.2. Marmot

We follow similar evaluation criteria to report results on the Marmot dataset in order to draw a direct comparison with the prior work. Our network separately detects the isolated and embedded formulas in a document image due to their variable sizes between isolated and embedded formulas.

Table 4. Performance comparison between our method and previous state-of-the-art approaches on the Marmot dataset.

Method	Formula	Correct (%)	Partial (%)	Total
Chu et al. [55]	Isolated	26.87	44.87	71.76
	Embedded	1.74	28.87	30.61
Phong et al. [11]	Isolated	50.37	39.14	91.18
	Embedded	22.9	58.45	81.35
Phong et al. [18]	Isolated	93	-	-
	Embedded	73	-	-
Ours (Without Pre-processing)	Isolated	92.5	4.64	97.14
	Embedded	80.6	6.23	86.83
Ours (Complete)	Isolated	93	4.86	97.86
	Embedded	81.3	6.77	88.07

summarizes the performance of our method on the Marmot dataset. As explained in Section 4.2.6, we calculate the accuracies of correct and partial detections. Our proposed mathematical formula identification system achieves the correct detection accuracy of $93\%$ and $92.5\%$ on isolated formulas with and without incorporating the pre-processing method, respectively. In embedded formulas, the system obtains the correct detection accuracy of $81.3\%$ and $80.6\%$ equipped with and without the proposed dilation method, respectively.

Besides calculating detection accuracy, we compute AP at an IoU threshold of 0.5 and 0.75 for both the isolated and embedded formula detection on the Marmot dataset. The achieved results are highlighted in Figure 10. Moreover, in Figure 11, we present the performance of correct detection accuracy over various IoU thresholds ranging from 0.5 to 1.0.

The qualitative performance analysis of the presented method on the Marmot dataset is exhibited in Figure 12, Figure 13 and Figure 14. We predict correct regions for 236 out of 253 formulas present in the test set in detecting isolated formulas. In the case of embedded formulas, the network is able to precisely detect 777 out of the 956 formulas from the test set.

Comparison with State-of-the-Art Methods

We compare our results with earlier approaches on the Marmot dataset in Table 4. From the table, It is evident that our cascade network with a deformable composite backbone has clearly outsmarted the prior state-of-the-art method [18] in detecting embedded formulas while accomplishing identical results in the case of isolated formulas. We reduce the relative error of 30% by achieving a detection accuracy of 81.3% on embedded formulas. Another point that is worth mentioning is the partial detection accuracy. The proposed system partly predicts 4.86% from the remaining 7% missing isolated formulas, which makes the total detection accuracy 97.86%. For embedded formulas, we achieve a partial detection accuracy of 6.77%, which adds up to an 88.07% total detection accuracy. Therefore, our network only missed 2.14% and 11.93% of isolated formulas and embedded formulas from the test set, respectively. The reduced number of missed detections in isolated and embedded formulas demonstrates the superiority of the proposed method.

5. Conclusions and Future Work

We introduce an end-to-end trainable network for the detection of formulas in document images. Our proposed method follows high-level architectural principles of traditional object detection approaches. Specifically, it exploits dilated document images fed into a Cascade Mask R-CNN equipped with a deformable composite dual backbone network. The proposed modifications help the network to achieve better generalization and detection performance. We achieve state-of-the-art performance on a higher IoU threshold with an f1-score of 0.917 on the ICDAR-2017 POD dataset. Furthermore, we reduce the relative error by 30% in detecting embedded formulas on the Marmot dataset with a correct detection accuracy of 81.3%. Not only do we improve the quantitative accuracy, but we also observe an outstanding improvement in terms of false-positive rates. Moreover, the presented work empirically establishes that without relying on heavy pre-processing pipelines, it is possible to achieve a state-of-the-art formula detection system in scanned document images.

For future work, we expect that a deeper backbone would be able to perform better in terms of both IoU and false positives. Moreover, the experiments can be extended to detect various graphical page objects such as figures, charts, titles, and headings in document images.