Artificially Intelligent Readers: An Adaptive Framework for Original Handwritten Numerical Digits Recognition with OCR Methods

Abstract

Advanced artificial intelligence (AI) techniques have led to significant developments in optical character recognition (OCR) technologies. OCR applications, using AI techniques for transforming images of typed text, handwritten text, or other forms of text into machine-encoded text, provide a fair degree of accuracy for general text. However, even after decades of intensive research, creating OCR with human-like abilities has remained evasive. One of the challenges has been that OCR models trained on general text do not perform well on localized or personalized handwritten text due to differences in the writing style of alphabets and digits. This study aims to discuss the steps needed to create an adaptive framework for OCR models, with the intent of exploring a reasonable method to customize an OCR solution for a unique dataset of English language numerical digits were developed for this study. We develop a digit recognizer by training our model on the MNIST dataset with a convolutional neural network and contrast it with multiple models trained on combinations of the MNIST and custom digits. Using our methods, we observed results comparable with the baseline and provided recommendations for improving OCR accuracy for localized or personalized handwritten text. This study also provides an alternative perspective to generating data using conventional methods, which can serve as a gold standard for custom data augmentation to help address the challenges of scarce data and data imbalance.

1. Introduction

We are seeing a rapid increase in the breadth and depth of adaptive artificial intelligence (AI) solutions to improve performance in areas such as computer vision for character recognition, which is receiving significant industry attention [1,2]. Optical character recognition (OCR) is an AI method that mimics the human-intelligence capability of visual recognition for the computational identification of machine and handwritten text and digits from a broad range of images containing text [3,4]. This includes identification of text or digits from images of typed text, handwritten text, or printed text into machine-encoded text, either from a scanned document as a pdf file, a picture of a piece of paper in png or jpeg format, or a scene photo with text in it, such as text on a coffee cup, text on the cover page of a book, or the license number on number plates. The number of researchers, academic centers and labs, and companies researching and developing computer vision and OCR solutions has risen significantly over the past several years [2]. Hence, an array of OCR solutions and tools are available widely. We discuss a few OCR applications in the context of our research below.

OCR is increasingly used for automation and enhancement of workflow and business processes. OCR is used by organizations to manage handwritten and printed text and document types identification and to parse data according to complex rules. OCR also has social and societal functional capabilities; for example, OCR can be used to provide an assistive solution for those with vision impairment by converting text images into text-to-speech [5]. One of the significant challenges faced by hospitals is the loss of patient medical information. OCR technologies can support electronic medical records and health information capture. Additionally, OCR can be used to automate information extraction from test results for input into hospital information management systems (HIMS), supporting the effective representation of patients’ medical histories. OCR tools such as Scayne and Expensify are used in finance, accounting and business process management for functions such as receipt recognition [6]. OCR is also used in loan processing, security and fraud detection, government sector services such as scanning legacy identification documents and automated license plate detection, and for supporting cultural and historical heritage initiatives, while fairly robust solutions are available for typed or printed machine-readable texts, automated handwriting recognition systems are yet to provide high levels of accuracy due to the challenges of variations in handwriting styles and composition of visual structure. Resolving these challenges remains an open problem, and improving handwriting OCR can lead to very valuable insights for decision support through the inclusion of handwritten data in the evaluation of public sentiment and other textual analytics solutions [7,8].

It is, thus, evident that OCR is an important technology and AI capability. It is, therefore, important to improve the accuracy of OCR tools with handwritten text, especially for specific handwriting styles, such as the handwriting of a particular doctor or a handwriting style from a particular region. Our work contributes to the body of knowledge of OCR methods by introducing the idea of fine-tuning MNIST with training data from a custom dataset belonging to a particular style to improve the accuracy for that specific style of handwritten digits. The model, method, and process used for OCR with such custom personalized or localized data can be extended to any handwritten style. Our work can also provide an alternative mechanism for generating custom data and conversion to MNIST format using conventional methods and can serve as a standardized mechanism for data augmentation to address the challenges associated with scarce data and data imbalance. The main contributions of this paper are as follows:

• We create a new handwritten custom dataset from a single contributor to ensure overall consistency of style.
• We train and apply a convolutional neural network to MNIST data for validation and then extend the method with custom data.
• Furthermore, we provide insights into the effects of training the model with variations in the proportions of custom data with and without rotations
• We provide a framework for adaptive OCR applicable to handwritten documents from a single source or belonging to a particular style of writing.
• Finally, we provide a comprehensive discussion of how the adaptive framework could be implemented and potential improvements with alternate technological solutions.

The rest of the paper is organized as follows. Section 2 presents the literature background of notable OCR methods. Section 3 mentions the problem statements and provides details of the dataset, and preprocessing strategies used for the experiments. Section 4 presents the stages involved in OCR and the architecture of the employed deep learning (DL) approaches. Section 5 sums up the total experiments performed and presents the experimental outcomes. Section 6 presents the analyses of the obtained results and the limitations of the employed methods. Finally, Section 7 provides the concluding remarks and the future research direction.

2. Related Works

In this section, we review the existing works related to OCR. OCR systems can be classified into two groups depending on the kind of input: handwriting recognition and machine-printed character recognition. OCR, to some extent, is considered a solved problem for standard machine text images. Machine-printed character recognition is a relatively straightforward problem since characters are typically of uniform dimensions, and their positions on the page can be predicted. The image input for this model can be in any format, such as JPEGs, PDFs, and documents. However, handwritten text recognition, a critical component of OCR, is still challenging. Unlike printed text, the wide range of human handwriting styles and the poor quality of handwritten text pose significant challenges in converting it to machine-readable language [9].

Handwritten Character Recognition plays a crucial role in the postal services industry by automating the process of address interpretation and facilitating efficient mail sorting. In earlier approaches, high-volume applications utilized task-specific readers to achieve high system throughput. These readers focused on specific fields of interest, taking advantage of the similarity in size and layout structure among similar documents. This enabled image scanners to extract the desired information efficiently, reducing image processing and text recognition time [10]. Address readers, another type of earlier system, were designed to locate the destination address block on mail pieces and read the ZIP code within that block. In cases where additional fields were read with high confidence, the system could generate a nine-digit ZIP code and apply a corresponding bar code on the envelope [11]. Modern systems in postal services have embraced advanced techniques, such as Hidden Markov Models (HMM), for handwritten address recognition. Kornai conducted a study on two variants of an HMM-based postal OCR system, one with pre-segmentation and one without. This study reported promising results on the CEDAR dataset, contributing to understanding handwritten address recognition [12]. Furthermore, region-specific use cases have emerged, such as recognizing Sinhala handwriting using OCR and image processing technologies [13]. These specific applications cater to the unique challenges of different languages and writing systems. Grid-based approaches have also gained attention in offline handwritten word recognition. Patel and Reddy explored the impact of a grid-based approach using Principal Component Analysis (PCA) for improved representation of handwritten Kannada words. Their study focused on recognizing district names within the Karnataka state, showcasing the applicability of grid-based methods in specific contexts [14], while significant progress has been made, challenges still exist in achieving high accuracy rates and adapting to varying handwriting styles. Ongoing technological research and development will continue to enhance the efficiency and effectiveness of postal services, ultimately improving the accuracy and speed of address interpretation and mail sorting processes.

Improvement of OCR accuracy for handwritten text remains an important issue for other industries as well, such as healthcare (handwritten medical notes by healthcare practitioners with varying styles), insurance (handwritten forms, mail envelopes, and handwriting from digital devices—stylus and touch screen handwriting), and banking (handwritten letters, checks, forms, and documentation, especially in developing and underdeveloped nations) [2,15,16]. In healthcare, OCR can be adapted to improve accuracy with individual medical practitioners. Similarly, OCR can be adapted in banking and insurance to enhance accuracy with individual users, clients, and executives in organizational processes with repetitive handwritten input, while postal services use OCR for handwriting, there is no need to customize it for the user. However, even postal services could customize based on the regional styling of characters, wherein the characters may vary within the same language based on regional grouping [17,18].

Handwritten text recognition systems are available as online and offline applications. Online systems work in real-time when users write the characters into a mechanism such as a touch screen or another touch-sensitive interface. They are structured differently since they can store time-based information, such as speed, velocity, number of strokes, and writing direction. Offline recognition systems, on the other hand, work with static data, i.e., bitmap images as input, and often require extensive pre-processing to perform these tasks. Consequently, the recognition task can be more challenging. More recent technologies based on the transformers architecture have shown promise in reading handwritten text, but the need to improve accuracy further remains [19].

The quality of OCR-generated text can impact various downstream natural language processing tasks, including NER and dependency parsing in certain workflows. Retrieval results on OCR-generated text should be viewed cautiously and manual supervision is recommended, as they can result in false positives. Topic models can also be affected by OCR quality in certain workflows, leading to increasingly divergent topics. It is, therefore, recommended to use high-quality OCR above 90 percent and better [20]. A new architecture for OCR-based image captioning called Long Short-Term Memory Relational Aware Pointer Network (LSTM-R) uses Long Short-Term Memory and a relation-aware pointer network to incorporate the geometrical relationships between OCR tokens. The relation-based pointer network copies words from OCR tokens based on their geometric relationship, and the model is optimized with multi-label loss. Experimental results demonstrated the effectiveness of LSTM-R over benchmarks, achieving state-of-the-art results on the TextCaps dataset [21]. It is also important to consider potential directions for developing post-OCR processing approaches. It is possible to create artificial materials to train machine learning models, utilize external resources for neural network-based models, improve error detection and correction tasks, and analyze performance with different OCR errors; it is also useful to focus on ‘handling errors’ involving word segmentation, consider ‘different weights for evaluation metrics’, create datasets with more detailed information, and develop ‘post-processing approaches for languages other than English’ [22]. Analysis by [23] revealed that most of the OCR software is incapable of accurately detecting variations in fonts and formats. Furthermore, while it can effectively convert text images, it fails to appropriately convert mathematical equations and symbols, providing only plain text as output. This highlights the need for new and improvised OCR tools to be developed in the future that can effectively perform the identification of mathematical equations and symbols [23].

According to [24], creating a dataset specific to a particular language and an associated period is highly beneficial for OCR correction workflows. Interestingly, using only a language-specific dataset based on the Bible created more errors than corrections. Ref. [24] considered this as being similar to a person from the biblical era attempting to fix OCR errors in modern texts. Furthermore, the error generation algorithm for Hebrew historical newspapers required only 105 manually corrected articles, which is significantly less than the amount of labeled training data needed for a neural network [24]. Ref. [25] showed that document AI and Textract had consistently lower error rates than Tesseract on various documents with different noise levels and fonts in English and Arabic.

Tesseract was more sensitive to noise and performed better only on noise-free documents in English. Further analysis by noise type showed that Textract and Tesseract performed better on grayscale images than on color images and struggled with blur, and Tesseract was more sensitive to salt and pepper noise. In conclusion, Document AI and Textract had better OCR accuracy than Tesseract, and accuracy varied across languages and noise types [25].

It is also important to take a quick view of some of the state-of-the-art solutions available currently. Popular OCR solutions include Tesseract, EASY OCR, Keras OCR, PyPDF2, DocTr, and TrOCR. Tesseract was initially proposed in [26]. Tesseract was one of the first architectures to handle white-on-black text with ease [26]. Tesseract was initially developed at HP in 1984 and 1994. Tesseract was open-sourced in 2005, then acquired by Google, and it works well with Tensorflow. Tesseract solves the problem of text localization, identifying where the text is in the document or an image [27]. Tesseract 4.00 has a new neural network subsystem as a text line recognizer. It has its origins in [28]. Easy OCR is the most straightforward way to implement OCR, and Easy OCR can be implemented with just a few lines of code. It is a Python package, and Jaided AI [29] maintains it. It was released in July 2020 and received a recent update in June 2022. EasyOCR uses Pytorch and OpenCV in the backend [30]. Keras OCR [31] provides end to end training pipeline to build new OCR models. It is a slightly better and packed version of Keras CRNN implementation and CRAFT text detection model. It provides a very accurate API for training a text detection and OCR pipeline [32]. Convolutional Recurrent Neural Network is a CNN, RNN, and CTC (Connectionist Temporal Classification) loss combination for image-based sequence recognition applications, such as scene text recognition and OCR [33]. The convolution is used to extract the feature sequence from an input image. The Recurrent neural network predicts the label distribution for each frame. Furthermore, a transcription layer translates the per-frame predictions into a final label sequence [34]. A convolutional neural network is used to construct Character Region Awareness For Text (‘CRAFT’) Detection to provide the character region and affinity scores. Individual characters are localized in the image using the region score, and each character is grouped into one instance using the affinity score [35].

PyPDF2 is a Pure-Python library that Fenniak [36] created as a PDF toolkit. It has the ability to extract document information [37], divide and merge documents page by page, crop pages, combine numerous pages into one, and encrypt and decode PDF files. DocTR was published by Mindee in March 2021 [38]. Its latest version was released in March 2022. It achieves end-to-end OCR in a two-stage approach; in the first stage, it implements text detection using Resnet 50 architecture, and in the second stage, it achieves text recognition by using CRNN. It can process both PDFs and images. It also detects rotated images. It can show text localization along with confidence. Its overall good implementation and gives overall accurate results [39]. More recently, we have seen the use of transformers for converting textual or handwritten data to machine-encoded text. The transformer concept and architecture were first introduced in the famous paper by [40], and since then, transformers have been increasingly used in NLP, computer vision, and other AI tasks. A Transformer architecture is a stack of encoder and decoder layers. Encoder and Decoder are composed of modules stacked on top of each other multiple times. The encoder will convert the English text into numerical representations. Those numerical representations are fed into a Decoder, and the decoder converts to desored output for further processing. Microsoft’s TrOCR models are encoder–decoder models with a text Transformer as the decoder and an image Transformer as the encoder [41]. The Hugging Face version of the TrOCR model is popular because of its longevity and simplicity. With many other implementations, it is observed that the same model or libraries cannot recognize handwritten and typed text with high accuracy. The TrOCR model can convert both text types into machine-encoded text. Furthermore, experiments have shown that the TrOCR model outperforms the current state-of-the-art models on printed and handwritten text recognition tasks.

Table 1. Overview of open-source OCR models.

Sr No.	Name	Brief Description	Authors	Format
1.	Tesseract	Tesseract is performing well for high-resolution images	[42]	Machine text
2.	TrOCR	It can produce very accurate results for machine and handwritten text and is based on Transformers	[41]	Handwritten and Machine text
3.	KerasOCR	Keras-OCR is designed for images. It produces decent results when text is embedded in a picture and the fonts and colors are not coordinated	[31]	Machine text
4.	DocTr	State-of-the-art performances on public document datasets, comparable with GoogleVision	[38]	Machine text
5.	Easy OCR	EasyOCR is a lightweight model that gives accurate results with organized texts like pdf files, receipts, and bills. It supports 80+ languages	[43]	Machine text
6.	PyPDF2	Can be used only for PDF and is not accurate in some cases.	[36]	Machine text

list the notable open-source OCR models.

3. Problem Formulation, Datasets, and Preprocessing

This section explains the problem statements tackled, the experimental dataset used, the process involved in generating Custom dataset, and preprocessing strategies implemented to prepare the input data for our experiments.

3.1. Problem Formulation

In this work, we explored the idea of developing an adaptive OCR framework and provide reports from our OCR experiments in working with custom data of a unique but consistent style from a single source in addition to baseline MNIST data. Thus, we contribute to addressing the challenge of improving the accuracy of OCR mechanisms with adaptive methods achieved through localized training of models with the inclusion of custom data in the training process.

3.2. Datasets

For our work, we have used two datasets, namely the Modified National Institute of Standards and Technology database (MNIST) and our own generated dataset. For ease of understanding, we termed our dataset as the Custom dataset.

• MNIST: The MNIST is a benchmarking [44] of 60,000 train handwritten digits and a test set of 10,000 (10 class labels) with each example represented as an image of 28 × 28 gray-scale pixels. Images are normalized to fit in a 20 × 20 pixel box while maintaining their aspect ratio. Images are centered in a 28 × 28 image. One column for each of the pixels because there are 28 × 28 pixels, there are 784 columns, and +1 for the labels. Pixel values are between 0 and 255, where 0 means black and 255 means white.
• Custom: Custom dataset consists of 240 training digits, 40 testing digits, and 20 validation digits, ensuring that the model is trained on a diverse set of data and can generalize well to new data points.
  The Custom dataset is also made available on our Githubrepository [45].

3.3. Custom Dataset Collection

For data collection, we have to create images of the digits. The following steps were taken to accomplish this. Since writing digits on paper and taking a photograph of them can be a challenging and time-consuming process, Microsoft Word version 10 was utilized to make images of the digits. The “draw with trackpad” option in the drawing tab of Microsoft Word was used for this purpose. Subsequently, screenshots of each digit were taken and grouped into batches of 10. Each batch contained digits from 0 to 9, with 10 digits per batch, and the name of the image was the gold label of what the digit represented.

• Preparing the Custom Dataset: Our Custom dataset consists of 300 handwritten digits collected using the method described in the data collection section. Two users wrote these digits, and we divided the dataset into three parts, each containing 100 digits. We then performed a stratified split on each of the three datasets to create training and testing sets. Specifically, we divided each set into 80 training images and 20 testing images, ensuring each category had an equal number of labels. This stratified split helped to prevent any bias in the model. Next, we concatenated the training images, creating a set of 240 training digits, and concatenated the testing images, creating a set of 60 digits. We then performed a stratified fold on the testing images, splitting them into sets of 40 and 20. The 20 digits were used as a validation dataset.

3.4. Preprocessing the Custom Dataset

To ensure consistency with the MNIST dataset, which has images of a fixed size of 28 × 28 pixels, we need to standardize our images to the same size. When digitizing a handwritten number, we followed several steps to ensure the accuracy and clarity of the resulting image. Firstly, the handwritten number image is loaded, and then, it is converted to a binary image format using the “L” mode, which only allows each pixel to be black or white. A white canvas of 28 × 28 is created the standard size for all MNIST images. The width or height of the image is then determined, and the image is resized such that the greatest dimension is 20 pixels, while the smaller dimension is scaled proportionally. To improve the image quality, the antialias function of pillow Library is applied, which smoothens jagged edges by averaging the colors of the pixels at the boundary and sharpening it. Subsequently, the resized image is pasted onto the 28 × 28 pixel white canvas, with 4 pixels from the top or side of the largest dimension used to center the picture. The smallest dimension is placed halfway between the original size of 28 and the scaled picture to center the image. After obtaining the pixel values of the new image, the values are adjusted to a range of 0 to 1, where 1 represents black and 0 represents white, using a formula to invert and normalize the values. As a result of these steps, the resulting image of the handwritten number is clear and accurate, making it easier to analyze and process. The original image (Figure 1b) was transformed into the MNIST-style image shown in Figure 1c using the process mentioned above. Figure 1a and our transformed digit Figure 1c are quite similar in appearance. However, our transformation process resulted in a sharper image compared to the original figure.

4. Materials and Methods

This section mentions the computational resource used to conduct the experiments, the phases involved in performing OCR, and the architecture of the employed CNN-based DL model. The computational resource used to perform the experiments is mentioned in

Table 2. Hardware resource specifications.

Item	Specification
CPU	Intel Core i3-7100 (-HT-MCP-) CPU @ 3.90 GHz
GPU	NVIDIA T4
CUDA	Version 10.1
OS	Ubuntu 17.10
Python	Version 3.10
Tensorflow	Version 2.12.0

.

4.1. Major Phases of Performing an OCR

The OCR workflow combines multiple phases, as described by [46].

• Source data input—this phase involves using an external optical device, such as a scanner or camera, to generate images that contain relevant alphanumeric text or symbols.
• Data preparation—covers various preprocessing operations, such as noise removal, to enhance source image quality and alignment of the image with the required standardized input format.
• Partitioning—multiple characters in the image are split into individual items in this phase so that a recognition engine can process the partitioned characters appropriately.
• Feature extraction—the partitioned characters are further “processed to extract different features.” The characters are then ‘recognized’ based on their alignment with these features.
• Classification—based on partitioning and feature extraction, the target image’s features are mapped to several categories and classified to appropriate values.
• Options—characters within source files can be classified using multiple techniques for pattern recognition using fundamental and statistical approaches.
• Validation and improvement—upon the completion of initial classification exercises, it may be observed that the results are rarely perfect. This is especially true for handwritten content with multiple authors, multifaceted fonts, and intricate languages. Quantitative, conceptual, linguistic, and probabilistic remedial approaches and fine-tuning can be performed to improve the accuracy of OCR systems.

4.2. Architecture of CNN-Based DL Model

CNNs have become popular for image classification tasks due to their exceptional ability to extract features from images [47]. Our study utilized the MNIST dataset, which comprises 70,000 handwritten digits from 0 to 9 represented in a 28-by-28-pixel grayscale format. To further enhance the dataset, we added our data to the original MNIST dataset, resulting in an augmented dataset used to train the CNN. The trained model was then evaluated on our custom test data to assess its accuracy and generalizability. The process flowchart with Custom data is shown in Figure 2.

For our experiments, we built a CNN Architecture, as shown in Figure 3, with the following parameters:

• In our model, we stack multiple Conv2D layers to extract increasingly complex features from an input image.
• Each Conv2D layer has parameters such as the number of filters, kernel size, padding, and activation function, which are tuned to optimize the performance of the network. The Conv2D layers are used to build feature maps from the data.
• The kernel size of the filters is usually (3 × 3), and we use padding to ensure that the filters fit the input image. We also use the ReLU activation function to introduce nonlinearity in our model. Max-Pooling is used to reduce dimensionality. In Max-Pooling, the output value is just the maximum input value in each patch (for example, the maximum pixel in a span of three pixels).
• The next step is to flatten out the output from the last pooling layer because the input of the fully-connected layer is not a 2D vector but a 1D vector. To flatten means to convert a 2D matrix into a 1D matrix.
• A fully-connected hidden layer is added to perform classification. The fully-connected layers combine the features extracted from the convolutional layers to create a model. Between fully-connected layers, dropout layers are added to remove specific neurons’ contributions to the following layer while leaving the rest intact. Dropout layers are applied to reduce overfitting.
• Finally, an activation function as softmax is used to classify the outputs as digits as 0, 1, 2 … 9.

The model information is presented in

Table 3. Model summary.

Layer (Type)	Output Shape	Number of Parameters
conv2d (Conv2D)	(None, 28, 28, 16)	160
max_pooling2d (MaxPooling2D)	(None, 14, 14, 16)	0
conv2d_1 (Conv2D)	(None, 14, 14, 32)	4640
max_pooling2d_1 (MaxPooling2D)	(None, 7, 7, 32)	0
conv2d_2 (Conv2D)	(None, 7, 7, 64)	18,496
max_pooling2d_2 (MaxPooling2D)	(None, 3, 3, 64)	0
conv2d_3 (Conv2D)	(None, 3, 3, 64)	36,928
max_pooling2d_3 (MaxPooling2D)	(None, 1, 1, 64)	0
flatten (Flatten)	(None, 64)	0
dense (Dense)	(None, 64)	4160
dropout (Dropout)	(None, 64)	0
dense_1 (Dense)	(None, 64)	4160
dropout_1 (Dropout)	(None, 64)	0
dense_2 (Dense)	(None, 10)	650

.

5. Experiments and Results

In this section, we describe the details of our experiments, the process, and the outcomes. For ease of understanding, we have grouped each experiment and its result. We have used the Accuracy and F-1 score [48] metrics below to measure the performance of our employed classification models given by the following equations:

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(1)

$$\mathrm{F}1=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(2)

where TP, FP, TN, and FN are True positive, False positive, True Negative, and False Negative, respectively.

5.1. Experiment 1

In this experiment, we trained the CNN Model on 60,000 MNIST images and tested the model on 10,000 MNIST test data points. The output from this experiment is not shown in

Table 4. Summary of experimental results with 40 Custom test digits.

Model	Train Data	Validation Accuracy	Misclassified Digits	Test Accuracy
CNN	MNIST	99.18%	5	87.5%
CNN	MNIST + 240 custom digits	98.91%	2	95%
CNN	MNIST + 960 custom digits	98.83%	2	95%
CNN	MNIST + 9840 custom digits	98.96%	2	95%
TrOCR	(Pretrained model)	NA	2	95%

because it was a standard experiment with MNIST data, and we only used it to validate our base process.

Results: Overall, the model performed well and generalized well to different data points, with very few mispredicted labels, as shown in Figure 4a. Furthermore, Figure 4b depicts the Loss vs. Epoch and Loss vs. Accuracy graph, demonstrating that the validation accuracy and validation loss were consistent with the training accuracy and training loss. The small gap between training and validation accuracy indicates that the model does not overfit and can generalize well. This model will serve as our baseline for comparison to other models. The model achieved an accuracy of 97.90% and a validation accuracy of 99.18%.

5.2. Experiment 2

In this experiment, the model was trained on a dataset of 60,000 images from the MNIST dataset and was then tested on our Custom test dataset consisting of 40 images. The MNIST test set, which consists of 10,000 images, was used as the validation dataset for this model.

Results: To evaluate the model’s performance on our Custom dataset, we used the same model as before and tested it on 40 images. The results showed that five of the 40 digits were misclassified, as seen in Figure 5. Notably, the digit “2” had the lowest accuracy of classification.

5.3. Experiment 3

The model was trained on a combined dataset comprising 60,000 images from the MNIST dataset and 240 images from our Custom dataset. To prevent overfitting and ensure that the model generalizes well, we used the MNIST test set, which contains 10,000 images, along with 20 digits from our Custom validation dataset, as the validation dataset for this model. After the training phase, we evaluated the model’s performance on our Custom test dataset of 40 images.

Results: The evaluation of the model’s performance on our Custom test dataset revealed that two of the 40 digits were misclassified, as shown in Figure 6a. Notably, the digits “4” and “5” were the ones that were misclassified. The graph shown in Figure 6b indicates that there is only a tiny gap between training and validation accuracy and loss, indicating that the model has maintained its generalizability and can fine-tune itself over our Custom dataset, achieving a low number of misclassification. The model achieved an accuracy of 97.07% on the combined dataset and a validation accuracy of 98.91%. These results suggest that the model performs well on both the MNIST and our Custom datasets, indicating that it has learned relevant common features across different datasets. These findings demonstrate the effectiveness of using a combined dataset and diverse validation data to improve the model’s performance on new data points.

5.4. Experiment 4

This model was trained on a combined dataset of 60,000 images from the MNIST dataset, 240 images from our Custom dataset, and 720 rotated digits. The rotated digits were generated by applying a random rotation angle between −15 degrees to 15 degrees to our 240 digits. For validation, we used the MNIST test set, which contains 10,000 images, and added 20 digits from our Custom validation dataset. Following the training phase, we evaluated the model’s performance on our Custom test dataset of 40 images.

Results: The evaluation of the model’s performance on our Custom test dataset revealed that two of the 40 digits were misclassified, as shown in Figure 7a. As seen in the graph presented in Figure 7b, there is a slightly increased gap between the training and validation accuracy, which was expected due to the addition of our large Custom dataset. Although the model’s generalizability decreased slightly, it still performed well on the Custom test dataset. The model achieved an accuracy of 96.30% on the combined dataset and a validation accuracy of 98.83%.

5.5. Experiment 5

This model was trained on a combined dataset of 60,000 images from the MNIST dataset, 240 images from our Custom dataset, and 9600 rotated digits. The rotated digits were generated by applying a random rotation angle between −15 degrees to 15 degrees to our 240 digits. For validation, we used the MNIST test set, which contains 10,000 images, and added 20 digits from our Custom validation dataset. Following the training phase, we evaluated the model’s performance on our Custom test dataset of 40 images.

Results: The evaluation of the model’s performance on our Custom test dataset revealed that two of the 40 digits were misclassified, as shown in Figure 8a. As seen in the graph presented in Figure 8b, there is a drastic increased gap between the training and validation accuracy, which was expected due to the addition of our larger Custom dataset. Although the model’s generalizability decreased slightly, it still performed well on the Custom test dataset. The model achieved an accuracy of 85.30% on the combined dataset and a validation accuracy of 98.96%.

5.6. Experiment 6

The model used in this experiment is based on the transformer architecture and was first introduced in the paper by Li et al. (2021) [41]. It was trained on the IAM dataset [49], and the specific model used in this study is a pre-trained model fine-tuned on the MNIST dataset accessible via [50]. On the MNIST dataset, the model achieved an accuracy of 99.52%. However, when tested on our dataset, it misclassified two digits, as seen in Figure 9.

5.7. Experimental Summary

We have provided the consolidated experimental outcomes of experiments 2 to 6 in Table 4.

6. Discussion and Limitations

Computer vision and digital image processing are crucial in multimedia, artificial intelligence, and robotics. Image analysis includes segmentation, feature extraction, and classification techniques. Human–computer interaction can make things easier for users, and optimal results with less computation time and multilingual character segmentation and recognition are possible. A segmentation-free approach using Deep Neural Network (DNN) is also possible in OCR, and this work may bridge the knowledge gap in automatic interaction between human–system and system–system interactions [51].

We use a CNN to recognize an image containing a single character. Text of arbitrary length is a sequence of characters, and such problems are solved using Recurrent Neural Networks, and LSTM [52] is a popular form of RNN [53]. Modernization of the Tesseract tool involved code cleaning and adding a new LSTM model. The input image is processed in boxes, line by line, and fed into an LSTM model. Even after a lot of training, Tesseract performs better, but it still needs to be improved to work on handwritten text and weird fonts. Additionally, rotated or skewed text may cause the Tesseract to malfunction.

One major limitation we faced during the model training was the scarcity of data, as we only produced about 300 digits. This limited our ability to make significant improvements to the model, which was already trained on a much larger dataset of 60,000 digits. To address this limitation and make further progress, we would need to increase our dataset size to at least 10,000 digits.

Additionally, although we used a CNN model for our digit recognition task, we acknowledge that transformer models such as TrOCR have demonstrated comparable accuracy, as seen in the above table. With the potential to capture more global representations of images compared to CNN models, fine-tuning a TrOCR-based model could yield even better results. Furthermore, while our smart-OCR research concept is demonstrated using digits, the final production-level application will be expected to extend the concept to handwritten textual artifacts, full texts, and documents corpora, including historical texts [54,55].

7. Conclusions and Future Work

In this paper, we demonstrated the viability of a simple schema to develop an OCR mechanism for creating an adaptive framework for custom digit recognition models, and its logical implication of flexibility of OCR models to specific writing styles. Once developed further with state-of-the-art neural networks, such as transformers for computer vision, this approach can be applied to a variety of industry-level use cases. This could include solutions in healthcare where individual medical practitioners have different writing styles, and in fraud detection to match or distinguish handwriting styles with greater accuracy. Given the tremendous potential for adaptive OCR applications, it is advisable to move adaptive OCR research to the forefront. We are hopeful that such adaptive OCR solutions would be an important part of the rapidly advancing artificial intelligence ecosystems worldwide. Currently, most NLP research and practice use machine-readable typed data and associated textual data distributions [56]. It would be very useful to develop OCR solutions for handwritten documents to create a seamless integration with NLP solutions, such as sentiment analysis and NLP-based socioeconomic modeling [57,58,59]. OCR is a mature discipline with industry-level solutions for identifying and ‘reading’ images of machine-printed text. However, due to the high degree of variations, OCR for handwriting recognition needs additional work. Based on the early-stage success of TrOCR, we believe there is significant potential for improving OCR solutions for handwritten text with transformer-based applications. We intend to explore the potential of fine-tuning and limited shot learning with pretrained transformer models to cater to user-specific digit recognition needs. Furthermore, the additional use of AI methods and tools to mimic human intelligence’s capability to identify text in varying colors, mixed sizes, and styles and other complex forms holds great promise. As an applied direction for future research, it is possible to use OCR methods to generate data for domains such as heritage culture and preservation from images of historical texts. One of the most important research areas to build upon would be the capability of an OCR application to be ‘flexible’ with custom handwriting styles. Incorporating the on-demand flexibility of OCR models would be a powerful way to advance the effectiveness of OCR with Custom data and variations in font styles. We also aim to incorporate world knowledge in the form of triples to address the domain adaptation challenges in identifying subtle sentiments [60].