SMSProcessing Using Optical Character Recognition for Smishing Detection ^†

Abstract

Instant messaging services are the main modern means of communication because they allow the exchange of messages between people anywhere and through many types of devices. Smishing involves sending text messages spoofing banks, government institutions, or companies in order to deceive. These messages often include malicious links that redirect users to fraudulent websites designed to steal personal information and commit financial fraud, identity theft, and extortion, among other crimes. Detecting smishing requires techniques to prevent access to dynamic links generated by cybercriminals to take control of devices or to consult blacklists of malicious links. Optical Character Recognition (OCR) recognizes text embedded in images without accessing links. This paper presents a conceptual model that uses OCR to extract text from messages suspected of smishing from a screenshot of a mobile device so that further processing can analyze whether it is smishing.

1. Introduction

Different devices such as smartphones, tablets, and computers do tasks such as sending text messages, tracking online purchases, providing technical support, and sharing multimedia content, increasing their use in work and daily life. The rise of text messaging applications has generated new forms of cyberattacks against users in Mexico. Smishing is a type of phishing attack carried out through instant messaging services and SMS messages; its name comes from the combination of the terms “SMS” and “phishing”. This variant of phishing uses messages that appear to come from a trusted sender, such as banks or government institutions, and often include links to fraudulent websites and are sent from fake phone numbers. Smishing detection tools usually validate links on blacklists or reconstruct them using machine learning (ML) and natural language processing (NLP). However, these techniques carry the risk of accessing dynamic links and directing users to fake websites, increasing the risk that they will unknowingly interact with fraudulent links [1], and cybercriminals could take control of their devices [2]. In Mexico, around 90% of users use instant messaging as a communication channel [3] and are victims of smishing attacks to obtain personal information or money, mainly through text messages and social networks that spoof legitimate financial institutions. Until November 2024, 174 financial institutions had been affected by identity theft, demonstrating that smishing is a significant challenge. Cybercriminals can avoid traditional security filters by modifying the content of messages or embedding text into images, which is difficult to detect automatically [4]. OCR is a technology for extracting text from images into an editable digital format; it automates the process of converting text embedded in images into structured data that can be automatically analyzed or processed [5,6]. Blanco-Medina et al. [7] propose a strategy that combines OCR and image preprocessing techniques to extract URLs from suspicious messages. This study focuses on URL analysis rather than message analysis to identify potentially malicious links in screenshots by combining computer vision and text recognition.

2. Materials and Methods

OCR is based on sequential steps to extract text from an image [8] and can be used as part of a solution to detect smishing without accessing malicious links. Each OCR step has its own mathematical principles and image processing techniques, which are described below as a proof of concept for text recognition in smishing messages:

• Conversion to grayscale. This reduces the complexity of the original image, preserving the luminance information and working with it instead of the three RGB color channels (red, green, and blue) using the following Equation (1) [9]:

  $$I(x,y)=0.2989R+0.5870G+0.1140B$$

  (1)
• Binarization. This is applied using a transfer function [10] to convert the image to black and white, as shown in Equation (2). For this process, a fixed threshold T is set for each pixel in the image.

  $$f(x,y)=\{\begin{array}{cc}255,& ifI(x,y)>T\hfill \\ 0,& ifI(x,y)\le T\hfill \end{array}$$

  (2)
• Morphological operations. This analyzes the shape and structure of objects in images using set theory, random functions, and lattice algebra to identify techniques such as dilation, erosion, opening, and closing [11].
• Character recognition. Models based on neural networks can be used for character recognition. Tesseract’s OCR engine (v. 5.5.0.20241111) uses LSTM (Long Short-Term Memory) models to recognize entire lines of text using recurrent neural networks [12].

3. Results

The proof of concept for the text extraction process from an image uses OCR with the ‘pytesseract’ library based on Tesseract OCR in Python 3.10 [13], which allows the text to be transcribed for further analysis. The text extraction process diagram is shown in Figure 1 and each of the stages are described below.

• Conversion to grayscale. The screenshot of the suspected smishing text message in Figure 2A is converted to grayscale, using the ‘cv2.cvtColor()’ function from the ‘OpenCV’ library, in order to reduce unnecessary information and work with a single channel for the binarization process (Figure 2B).
• Background detection. Background detection. A fixed threshold is used in screenshots of messages with a light background because they have a high contrast between the text and the background, which facilitates the binarization process. In images with a dark background, the contrast is lower, so it is necessary to invert colors using the ‘cv2.bitwise_not()’ function. To identify whether a theme is light or dark, the average brightness of the image pixels is calculated with ‘np.mean()’ using the ‘NumPy’ library, and a threshold of 57 is set. If this threshold is exceeded, the background is identified as dark.
• Binarization. When a dark background is detected, an adaptive threshold is applied with ‘cv2.adaptiveThreshold()’ to the inverted grayscale image to obtain a binarized image. If a light background is detected, a fixed threshold of 127 is applied to the grayscale image (Figure 2C) using the ‘cv2.threshold()’ function, and the binarized image is obtained (Figure 2D).
• Extracting text from the binarized image. The function ‘pytesseract.image_to_string()’ is applied to the binarized image to extract visible text from the image and return it as a text string, as shown in Figure 3.
• Sanitization of extracted text. The text extracted from the image is sanitized for further analysis. The text is converted to lowercase using ‘lower()’ and line breaks and extra spaces are removed using ‘replace()’ and ‘strip()’. The transcription of the text is shown in Figure 4.

4. Discussion

The text extracted from the image does not contain the link information that smishing messages often contain due to the image processing, which is lost during the binarization stage because of the characteristic blue color of links, and the proposal of this research is not affected. OCR can help detect smishing because it extracts the rest of the text in messages that can be analyzed based on keywords, in this case in Spanish.

Tests were performed with screenshots in light mode and dark mode, simulating the settings that a user’s mobile device might have, to determine the threshold and verify that character recognition is not affected by the loss of information from the suspected smishing message, and so far it has shown good qualitative results. Currently, tests are being conducted with an image database to determine the quantitative effectiveness of OCR.

5. Conclusions

Implementing OCR for smishing detection helps users to take a screenshot and ensure that they do not interact with the suspicious message, without the need to copy the message or link, thus avoiding the risk of being redirected to fake websites or executing malicious code on devices.

Sanitized text recovered from a screenshot of the suspicious smishing message requires additional techniques to complement the smishing detection phase, such as machine learning or natural language processing techniques that help contextualize the message received in Spanish.