Application of Binary Image Quality Assessment Methods to Predict the Quality of Optical Character Recognition Results

Abstract

One of the continuous challenges related to the growing popularity of mobile devices and embedded systems with limited memory and computational power is the development of relatively fast methods for real-time image and video analysis. One such example is Optical Character Recognition (OCR), which is usually too complex for such devices. Considering that images captured by cameras integrated into mobile devices may be acquired in uncontrolled lighting conditions, some quality issues related to non-uniform illumination may affect the image binarization results and further text recognition results. The solution proposed in this paper is related to a significant reduction in the computational burden, preventing the necessity of full text recognition. Conducting only the initial image binarization using various thresholding methods, the computation of the mutual similarities of binarization results is proposed, making it possible to build a simple model of binary image quality for a fast prediction of the OCR results’ quality. The experimental results provided in the paper obtained for the dataset of 1760 images, as well as the additional verification for a larger dataset, confirm the high correlation of the proposed quality model with text recognition results.

1. Introduction

The growing availability and popularity of mobile devices and embedded systems equipped with visual sensors increase the interest in onboard computations, as well as the development of computation methods and models useful for fast processing using limited hardware resources. Such limitations, related to memory amount and computational power, often hinder the real-time image analysis that may be important in scenarios where another image may be captured after changes in some camera parameters or its location. Some examples of such systems may be related to video-based navigation in mobile robotics, video inspection and diagnostics, and optical text recognition.

Images captured by cameras integrated with mobile devices are usually subject to various distortions, particularly considering their acquisition in unknown lighting conditions, including non-uniform illumination, presence of shadows, etc. Although in many applications related to object detection and classification, image segmentation, traffic monitoring, video surveillance, object recognition and tracking, etc., many such distortions may be efficiently eliminated or their influence is not critical, there are some areas where the analysis of natural images may be significantly influenced by them.

An obvious example may be a non-uniformly illuminated document image containing many small details—alphanumerical characters in such a case. Since a typical image preprocessing operation in Optical Character Recognition (OCR) systems is the binarization of input images, its results strongly depend on the presence of shadows and some other illumination issues. Application of the simplest global thresholding methods, such as Otsu [1], does not lead to satisfactory binary images in such cases, affecting not only the readability of some characters but often converting some parts of the image, even containing some characters, into solid black areas or a white background. In some cases, even the application of more advanced adaptive binarization methods does not lead to satisfactory results ensuring correct text recognition during further processing stages.

In some situations, users can capture another photo of a document image, but the problem lies in the relatively long time necessary for the OCR procedure, particularly if text recognition is applied as a cloud service with an additional transmission required. Therefore, a convenient solution would be the image quality assessment (IQA) conducted with a metric highly correlated with OCR accuracy, which might be calculated much faster than the execution time of the text recognition process. One of the possible solutions might be the application of no-reference (“blind”) IQA methods for input images in a similar way to the approach proposed for the fault-tolerant video-based OCR in the paper [2].

Nevertheless, the computation of such metrics for color or greyscale input images, although less demanding than the OCR procedure, is still much slower than similar calculations for binary images. Furthermore, the text recognition results depend on the binarization method [3,4,5] that is applied between image quality assessment of such greyscale or color images and the OCR procedure. Therefore, a more appropriate solution is the application of the binary image quality assessment for the selected binarization method. Unfortunately, the development of IQA methods for binary images is not as advanced as for greyscale or color images where a variety of full-reference (FR), reduced-reference (RR), and no-reference (NR) metrics [6] have been proposed during recent years.

Although some quality metrics have been proposed also for binary images [7,8,9,10], they are based on the comparison of a binary image with the “ground-truth” (reference) image. Such a full-reference approach cannot be directly applied for unknown images captured by a camera due to unavailability of reference images. Therefore, the idea proposed in this paper is based on the mutual comparison of the image binarization results using selected thresholding methods. It is assumed that a high similarity between two binary images occurs for two high-quality images rather than for low-quality binarization results. Therefore, such high similarities should be correlated with satisfactory OCR results. Nevertheless, an appropriate choice of the binary image quality metric used during these computations requires an additional verification using a dedicated image dataset containing various document images captured in non-uniform illumination subject to several binarization methods, discussed in the subsequent parts of this paper.

2. Proposed Approach and Its Verification

2.1. Mutual Similarity of Binary Images

The proposed idea of a fast prediction of the OCR results’ quality based on the application of binary image quality assessment makes it possible to significantly reduce the computational burden and prevent the text recognition for low quality images containing some areas with unreadable characters. It utilizes the mutual comparisons of binary images obtained using various thresholding methods; however, the comparisons may be conducted using various methods.

According to the assumptions related to the highest mutual similarity of high-quality binary images, the experimental verification of the idea has been conducted using the designated dataset based on the WEZUT OCR Dataset [11], which contains 176 non-uniformly illuminated document images with the commonly used placeholder text “Lorem ipsum”. The WEZUT OCR dataset is available at https://okarma.zut.edu.pl/?id=dataset (accessed on 12 October 2024). All images have been captured by a Digital Single-Lens Reflex (DSLR) camera Nikon N70 and their size is $2861\times 1965$ or $2872\times 1940$ pixels. This dataset has been extended using various image binarization algorithms, further applied instead of the default first step of the OCR algorithm. Finally, the dataset of binary images consisting of 1760 images has been created as a result of the application of the following adaptive image binarization methods: Bernsen [12], Bradley [13], Feng [14], Meanthresh, NICK [15], Niblack [16], Sauvola [17], and Wolf [18]. Additionally, two methods based on deep learning proposed by Liedes (the code is available at https://github.com/sliedes/binarize—accessed on 12 October 2024) and Masyagin (ROBIN—RObust document image BINarization tool—the code is available at https://github.com/masyagin1998/robin—accessed on 12 October 2024) have been used. Obviously, global image thresholding methods have not been applied due to their inadequacy for non-uniformly illuminated document images. The illustrations of some binarization results obtained for a representative sample image are presented in Figure 1, Figure 2, Figure 3 and Figure 4.

All 1760 images have been used as inputs for the open source Tesseract OCR engine (version 5.0.0) available at https://github.com/tesseract-ocr/tesseract (accessed on 12 October 2024), originally developed by Hewlett-Packard, further supported by Google, and currently evolving thanks to a group of developers led by Ray Smith. The OCR engine has been used without dictionary and language support to prevent their influence on the obtained results, replacing its default binarization with the individual methods mentioned above. The obtained text recognition results for each image have been compared with the “ground-truth” text provided in the WEZUT OCR Dataset and then the Levenshtein Distance (LD) between those two strings has been calculated. The computation of the classical Levenshtein Distance, also known as edit distance, is based on the counting of the number of simple edits necessary to convert one string into another. The operations considered as simple edits are character insertion, deletion, and substitution.

Since the WEZUT OCR Dataset contains photos of the documents printed using five different popular font shapes (Arial, Times New Roman, Calibri, Courier, and Verdana) with some typical modifications of attributes (normal, bold and italic versions of all fonts, as well as bold italics), each of the 1760 images obtained after thresholding has been compared with their equivalents obtained from the same source image using the other binarization methods. Finally, the nine similarity values have been averaged, leading to an aggregated mutual quality index. Then, the Pearson’s linear correlation coefficient values between them and the Levenshtein Distances computed for the same image have been determined to verify the appropriateness and usefulness of each image quality/similarity metric for the prediction of the quality of OCR results. Obviously, as it may be noticed in Figure 1, Figure 2, Figure 3 and Figure 4, some binarization methods have caused the loss of data and some characters are impossible to recognize.

During the experiments, some classical metrics based on the confusion matrix, typically used for the evaluation of binarization methods, as well as for performance metrics for pattern recognition or classification purposes, have been used, mainly due to their fast computation. The most widely known such metrics include Precision (defined as the ratio of true positives to all predicted positives) and Recall/Sensitivity (ratio of true positives to all actual positives), where black pixels representing the foreground information are considered positives and white background pixels negatives. The other metrics, based on similar computations, are F-Measure (harmonic mean of Precision and Recall), Accuracy (ratio of the sum of all true positives and true negatives to all pixels), pseudo-Precision, pseudo-Recall, pseudo-F-Measure, and Specificity [19], as well as the less popular Peak Signal-to-Noise Ratio (PSNR), Balanced Classification Rate (BCR), S-F-Measure, and Geometric Accuracy.

Nevertheless, the evaluation of image binarization results may also be conducted using some other approaches considered in this paper. The first interesting idea is the application of the Border Distance proposed by Zhang et al. [10]. This method is based on the assumption that the Border Distance of the modified pixels plays an important role in the binary image quality evaluation and the larger distances are related to worse image quality due to their larger influence. Since the distance may be defined in various ways, three approaches are considered based on the chessboard, city block, and Euclidean distances, leading to three types of the BDPSNR metric.

Another approach to binary image quality evaluation is the application of the Distance Reciprocal Distortion (DRD) metric [7]. This metric also utilizes the observation that the quality perception of document images is dependent on the distance between two pixels. Therefore, it is assumed that the influence of the horizontal and vertical neighbors is higher than the diagonal ones. All pixels differing between two compared images are then weighted by the reciprocal of a distance to the central pixel using the $5\times 5$ pixel weighting matrix. Then, the local DRD values are computed for the neighborhoods of the flipped pixels, further aggregated, and divided by the number of $8\times 8$ pixel blocks containing at least two differing pixels in the reference image (fully white and fully black blocks are excluded).

The last considered binary IQA metric is the Misclassification Penalty Metric (MPM), proposed in the paper [8], based on the idea of the penalization of the misclassified pixels depending on distances from the border of the reference objects. After the division of the obtained result by the sum of all pixel-to-contour distances in the image, the normalized value of the metric is obtained.

The approach proposed in the paper is based on the application of several of the above metrics for the calculation of mutual similarities between pairs of binary images obtained from the same color or greyscale input image. Since most of the considered thresholding methods belong to the family of adaptive binarization algorithms, one may expect quite similar results obtained even for non-uniformly illuminated images. Therefore, it is proposed to calculate mutual similarity values pairwise for each combination of two binary images originating from the same input image applying the selected metrics. The most appropriate binarization methods should not produce images strongly differing from the results obtained using the other thresholding algorithms; hence, the final OCR result’s quality should also be better than that achieved using the other binarization methods.

Since no-reference quality metrics for binary images are not known and the binary image quality assessment is based on the comparison with ground-truth images, it is proposed to determine the mutual similarities assuming that one of the binary images is considered a pseudo-reference image. The average mutual similarity values obtained as the results of such comparisons with images obtained the other binarization methods are saved as the final metrics separately for each of the considered elementary metrics. The illustration of this idea is presented in the form of a simplified flowchart in Figure 5.

The first stage of the experimental verification of the proposed idea has been related to the calculation of the correlation values for each of the above discussed metrics, applied for the computations of the aggregated mutual similarity and the Levenshtein Distances obtained for each of the 1760 images.

2.2. Combined Metrics and Discussion of Results

The experimental results presented as the correlations of individual metrics used for the computation of aggregated mutual similarities with Levenshtein Distances obtained for all 1760 images are shown in

Table 1. Pearson’s linear correlation coefficient (PLCC) of individual IQA metrics with the Levenshtein Distance for the considered 1760 images. The two best results are marked with bold font.

Quality Metric	PLCC for Tesseract
Precision	0.5594
Recall/Sensitivity	0.1003
F-Measure	0.8460
Pseudo-Precision	0.5972
Pseudo-Recall	0.1127
Pseudo-F-Measure	0.8810
Specificity	0.2957
Balanced Classification Rate (BCR)	0.1397
S-F-Measure	0.2019
Accuracy	0.2590
Geometric Accuracy	0.2165
Peak Signal-to-Noise Ratio (PSNR)	0.2543
Border Distance (chessboard) [10]	0.2939
Border Distance (city block) [10]	0.2805
Border Distance (Euclidean) [10]	0.2932
Distance Reciprocal Distortion (DRD) [7]	0.3662
Misclassification Penalty Metric (MPM) [8]	0.1928

. As it may be noticed, for some of the metrics, correlation values are very small, and therefore they cannot be considered useful for predicting the quality of final OCR results. The best results have been obtained for F-Measure and pseudo-F-Measure; hence, these two metrics are the most appropriate for the analyzed application.

Considering the encouraging results obtained by the combination of metrics presented in some earlier papers, including their application to binary images [20], the next conducted experiments have been related to the validation of the combination of the two abovementioned “best” metrics. The first model is defined as the weighted product of two metrics:

$$C{M}_1=\prod _{i=1}^2{Q_i}^{w_i}={(\mathrm{F}-\mathrm{Measure})}^{w_1}·{(\mathrm{pseudo}-\mathrm{F}-\mathrm{Measure})}^{w_2},$$

(1)

where $Q_i$ are the averaged mutual similarities obtained using the individual metrics being combined and $w_i$ are their weights obtained as the result of optimization using MATLAB’s (version R2024a) fminsearch function.

The second considered model is defined as the weighted sum of two individual metrics, also used as the averaged mutual similarities:

$$C{M}_2=\sum _{i=1}^2\left(a_i·{Q_i}^{w_i}\right)=a_1·{(\mathrm{F}-\mathrm{Measure})}^{w_1}+a_2·{(\mathrm{pseudo}-\mathrm{F}-\mathrm{Measure})}^{w_2}.$$

(2)

The main motivation of the use of the above defined combined metrics originates from the application of a similar approach for natural images for which a significant increase in the correlation between the objective and subjective quality scores has been reported. Some initial experiments have been conducted for the nonlinear combination of three metrics, MS-SSIM, VIF, and R-SVD, further changed into the combination of MS-SSIM, VIF, and FSIM_c, leading to CISI metric [21], which is highly correlated with the subjective quality scores available in image quality assessment (IQA) datasets.

Further extensions and applications of this approach for many various types of images, including binary images [20], remote sensing, or stitched images, fully confirmed the validity of this approach. A similar approach has also been applied by Netflix researchers in their Video Multi-Method Assessment Fusion (VMAF) metric [22]. Particularly interesting results have been obtained applying the combined metric based on the weighted sum, according to Formula (2), for multiply distorted images, proposed in the paper [23]. Although some attempts at the use of neural networks for designing the combined metrics have also been made [24], leading to comparable performance, the use of simpler combinations is more convenient for the considered application.

As it may be seen in

Table 2. Pearson’s linear correlation coefficient (PLCC) values of two combined IQA metrics with the Levenshtein Distance for the considered 1760 images together with the optimized weights for the combined metrics.

Metric	PLCC	${\mathit{a}}_1$	${\mathit{a}}_2$	${\mathit{a}}_3$	${\mathit{w}}_1$	${\mathit{w}}_2$	${\mathit{w}}_3$	${\mathit{w}}_4$
CM1	0.8960	−	−	−	$-0.9643$	$2.3779$	−	−
CM2	0.9077	$0.9098$	$0.0537$	−	$-0.7213$	$0.6592$	−	−
CM3	0.9085	$1.1595$	$-0.9666$	$-1.1790$	$1.0414$	$-0.1376$	$1.0231$	$0.0157$

, the combination of two metrics has led to a noticeable improvement in correlation between the obtained IQA values and Levenshtein Distances, particularly for the weighted sum denoted as $C{M}_2$. Further experiments related to the additional use of the third metric in both combinations have not led to significant improvements of the computed PLCC values, including the use of neural networks, as well as the extension of the proposed approach for video sequences. Additionally, a simple combination of the abovementioned two models, leading to the combined metric $C{M}_3$, has also been considered, which may be expressed in the following form:

$$C{M}_3=a_1·{Q_1}^{w_1}+a_2·{Q_2}^{w_2}+a_3·{Q_1}^{w_3}·{Q_2}^{w_4},$$

(3)

where $Q_1$ is the F-Measure and $Q_2$ is the pseudo-F-Measure in this case. Nevertheless, its application has led to a small improvement of the achieved correlation to 0.9085, as shown in Table 2.

To illustrate the properties of the proposed combined metrics based on the averaged mutual similarities of binary images, a sample image from the WEZUT OCR Dataset and its three binary versions are presented in Figure 6, together with the values of three combined metrics and the respective values of the Levenshtein Distance. The relations between the visual quality of the binary images, Levenshtein Distance values, and the results obtained using the three proposed combined metrics may be easily noticed. The best results among three presented methods may be observed for the NICK thresholding.

3. Additional Experiments

To verify the universality and usefulness of the proposed approach, some additional experiments were performed for a larger dataset of images, being the results of application of 12 binarization methods for the document images captured by mobile phone cameras. The algorithms selected during these experiments were Otsu [1], Bernsen [12], Niblack [16], Sauvola [17], Wolf [18], Gatos [25], NICK [15], Baitaneh [26], Singh [27], Su [28], WAN [29], and ISauvola [30].

As the source data (nearly 13 GB) for the binarization algorithms, the images from the SmartDoc-QA dataset available at https://zenodo.org/records/5293201 (accessed on 12 October 2024) were chosen due to the presence of several single and multiple distortions [31]. The SmartDoc-QA dataset contains two subsets with 2130 images of 30 documents each, captured using Nokia and Samsung smartphones, respectively, under varying conditions (subject to motion blur, out-of-focus blur, differing light conditions, and perspective angles), together with ground-truth data containing text transcriptions (the expected OCR results). The three types of documents present in the dataset consist of contemporary documents, old administrative documents, and shop receipts (10 samples per type). Both subsets have different size of images: $3264\times 2448$ pixels for Nokia and $4128\times 3096$ pixels for Samsung.

For the verification of the proposed approach, all images from the SmartDoc-QA dataset were binarized using the 12 abovementioned algorithms, and mutual comparisons were conducted similar to the WEZUT Dataset used in previous experiments. Nevertheless, since most of the images contain relatively large background areas that produce solid white or black fragments of binary images, depending on the individual thresholding method, one cannot expect as high mutual similarities as for the WEZUT Dataset containing document images without large background areas. An illustration of this phenomenon is presented in Figure 7, where the sample image from the SmartDoc-QA dataset is shown together with its results of binarization using three various methods. The easily noticeable differences in the background areas of the binary images are the main reason for the decreased correlations between the results of the binary image quality assessment and the Levenshtein Distance for this extended dataset.

Furthermore, to verify the universality of the proposed approach, apart from the Tesseract engine, the other OCR software implemented in Python by JaidedAI, known as EasyOCR (version 1.7.2), was used. Its code is available on GitHub at https://github.com/JaidedAI/EasyOCR (accessed on 12 October 2024). Therefore, the binary IQA metrics were computed mutually for each binarization algorithm and the 11 others, and then their correlations with the Levenshtein Distances (LD) obtained after applying the Tesseract and EasyOCR engines independently were determined. Finally, the results achieved for various binarization methods were averaged, leading to 2130 average correlation coefficients for each of two subsets (Nokia and Samsung) between the IQA metric and the LD values calculated separately for two OCR engines. The correlations obtained for individual metrics are presented in

Table 3. Averaged Pearson’s linear correlation coefficients (PLCC) of individual IQA metrics with the Levenshtein Distance for the considered images from the SmartDoc-QA subsets. The highest values in each column are marked with bold font.

OCR Engine	Tesseract		EasyOCR
Quality Metric	Nokia	Samsung	Nokia	Samsung
Precision	0.2275	0.2966	0.2697	0.3151
Recall/Sensitivity	0.3539	0.3286	0.4111	0.4385
F-Measure	0.4811	0.4569	0.5480	0.5490
Pseudo-Precision	0.3571	0.1585	0.3814	0.1935
Pseudo-Recall	0.2293	0.2914	0.2719	0.3922
Pseudo-F-Measure	0.4523	0.4865	0.5170	0.5756
Specificity	0.3402	0.4343	0.3634	0.4932
Balanced Classification Rate (BCR)	0.3829	0.2832	0.4476	0.3857
S-F-Measure	0.4272	0.3829	0.4835	0.4832
Accuracy	0.4789	0.3729	0.5073	0.4021
Geometric Accuracy	0.4169	0.3807	0.4737	0.4761
Peak Signal-to-Noise Ratio (PSNR)	0.2336	0.3603	0.2580	0.3559
Border Distance (chessboard) [10]	0.2129	0.3253	0.2386	0.3191
Border Distance (city-block) [10]	0.2130	0.3273	0.2395	0.3197
Border Distance (Euclidean) [10]	0.2134	0.3254	0.2391	0.3190
Distance Reciprocal Distortion (DRD) [7]	0.4536	0.3534	0.4878	0.3782
Misclassification Penalty Metric (MPM) [8]	0.0655	0.1453	0.0748	0.1264

, where the highest values may be observed for F-Measure and pseudo-F-Measure, similar to previous experiments conducted for the WEZUT Dataset.

Having confirmed the appropriateness of the choice of elementary metrics, the last step of the verification of the proposed approach concerned the calculation of the combined metrics based on the F-Measure and pseudo-F-Measure for the extended dataset of 4260 binary images originating from the SmartDoc-QA database. The obtained values of the averaged PLCC obtained for all three considered metrics are presented in

Table 4. Averaged Pearson’s linear correlation coefficients (PLCCs) of the combined metrics with the Levenshtein Distance for the considered images from the SmartDoc-QA subsets.

OCR Engine	Tesseract		EasyOCR
Quality Metric	Nokia	Samsung	Nokia	Samsung
$C{M}_1$	0.4839	0.5002	0.5569	0.6002
$C{M}_2$	0.4855	0.5003	0.5669	0.5996
$C{M}_3$	0.4892	0.5005	0.5734	0.6037

separately for the Tesseract and EasyOCR engines, as well as for the Nokia and Samsung subsets.

Comparing the best results achieved for the elementary metrics marked with bold fonts in Table 3 to the results reported in Table 4, the increase in the Pearson’s correlations with the Levenshtein Distances may be observed, similar to the WEZUT OCR Dataset. This observation confirms the validity of the proposed approach, also using some other binarization methods for a more demanding database.

4. Conclusions and Further Research

The proposed method for predicting the quality of OCR results, based on the application of image quality assessment of binary images as the aggregated mutual similarity indexes makes it possible to prevent the necessity of time-consuming text recognition, particularly for low-quality images. Due to a much faster execution of adaptive binarization, which may be efficiently executed in parallel also using integral images to speed up the computations, such as, e.g., in the Bradley method [13], after additional quality assessment of the results, mobile device users may obtain the information about the necessity for the acquisition of another image. Since some OCR engines work as cloud services, there is no necessity of sending various binary versions of the input image to the cloud using the proposed approach. In this case, the transmission time may be considered as an additional constraint, dependent on the network properties. The proposed preprocessing may be useful, particularly for devices where the execution of the OCR engine would be troublesome, e.g., due to memory limitations. The main assumption is the most appropriate preprocessing of the input image before a single run of the OCR engine leading to the best possible text recognition results.

The experimental verification of various binary IQA methods confirmed the usefulness of the F-Measure and pseudo-F-Measure for this purpose, as well as the advantages of their nonlinear combinations, leading to the linear correlation between the proposed combined aggregated mutual similarity and Levenshtein Distances of nearly 0.91 for the WEZUT Dataset. Noticeably smaller correlation values observed during the additional verification for the larger dataset are caused by the presence of some differences in the binarization of background areas using individual thresholding methods. Since most of the metrics are based on the mutual comparison of pixels, these differences influence the binary image quality assessment results regardless of the final OCR effects, decreasing the overall correlation of the binary IQA and the OCR results. Nevertheless, these experiments fully confirmed the choice of elementary metrics, as well as the validity and usefulness of the proposed approach. During further research, it is planned to develop even better models utilizing the combination of metrics also for short video sequences.