Text Summarization Using Deep Learning Techniques: A Review ^†

Abstract

The process of text summarization is one of the applications of natural language processing that presents one of the most challenging obstacles. This is one of the most challenging duties since it demands an in-depth understanding of the information that is being retrieved from the text; as a result, it is one of the most time-consuming as well. Traditional methods of paraphrasing a text each come with their own individual set of restrictions; this is why it is vital to develop new methods in order to achieve better results in paraphrasing a text. Deep learning has been used, which has resulted in a paradigm shift in the way natural language processing is carried out. The tremendous progress that has been made in the fields of sentiment analysis, text translation, and text summarization can be attributed to the application of methodologies that are based on deep learning. The utilization of these various approaches, which resulted in the production of these advancements, is a primary cause of these breakthroughs. We have outlined a variety of deep learning procedures with the goals of summarizing texts and analyzing details in order to prepare these methods for possible applications in future research.

1. Introduction

Day-to-day increases in information pose a problem for information analysis. Therefore, finding relevant data becomes costly and time consuming, as does determining which data are of interest. Text summary is the answer to providing information that is clear and useful. We explore a number of text summary strategies in this manuscript, as well as the many approaches those have been tested. Gambhir et al. [1] provided a very detail discussion on classical text summarization techniques. Extractive summarization is covered with respect to the approach, techniques, and type of summary. Depending on the characteristics employed, the document number, the method used, and other considerations, there are many forms of text summary.

Table 1. Different types of summaries based on various factors [1].

Sr. No.	Summaries Depending on Various Criteria’s	Parameters
1	One or many input documents	Various types of text data documents
2	Some words, new words method of text summarization	Outcomes based on semantics
3	Query-focused, generic	Purpose
4	Unsupervised, supervised, and semi supervised	Data Availability
5	Monolithic, multiple, and cross-lingual	Language-Dependent

[1] lists of all the relevant factors in detail. There are extractive and abstractive methods for summarizing text. Under the heading of abstractive text summarizing, Gupta et al. [2] addressed particularly thorough text summary algorithms. More difficult than extractive text summarization is abstractive text summarization. Depending on various factors, there are various approaches to implement abstractive text summarization. The many categories of abstractive text summarizing approaches are elaborated in Figure 1 [2].

2. Literature Survey

This part of the manuscript covers a variety of deep-learning-based techniques for performing text summarization. Some models are used on other languages, such as Bengali, Vietnamese, and Arabic. Vietnamese text summarization using supervised learning was proposed by Thu et al. [3], and is based on neural networks. The method divides words into two sets: nouns and other word sets. The process reduces the matrix’s dimensions. Consequently, the feature section may be useful. Three-layer feed-forward neural networks were employed by Thu [3]. Their own corpus of Vietnamese text has been generated. Due to the lack of prior work on Vietnamese text summarizing, a baseline technique was used for comparison analysis. The utilized algorithm is typical for the English language. Vietnamese text summary does not employ any language-specific techniques. An opposition base learning (OBL) approach used to enhance the evolutionary search with quality enhancement was introduced by Abuobieda et al. [4]. OBL uses an evolutionary algorithm (EA) to boost performance. EA saves the intermediate state during computation. Such states can be used by OBL to understand the search space behavior. The proposed method generates the optimal solution. Performance tuning in the evolutionary algorithm increases the accuracy compared with the random algorithm. They focus only on the initial population of the DE algorithm. Kabeer et al. [5] offered both traditional and graph-based techniques to construct summaries for Malayalam documents. The statistical technique is used to analyze and give weight to a sentence to rank it. The graph-based technique is used to extract the semantics of the sentence from a set of words. In the first phase of graph generation, extraction of the subject, predicate, and object is carried out. Graphs are generated on the basis of these three key parameters. Sub-graphs are generated using these three components to generate the summary. A logistic-regression-based model was used by Hong et al. [6]; they called it RegSum. In the first phase, a regression-based approach is used to identify the keyword. The weight of a word is determined by taking into account its location, position, type, and relative rank. The set of words present in the whole human-generated summary is labelled as gold standard keywords. The determinant point process (DPP) outperforms the method on R1 Score. Fatteh [7] introduced a hybrid model that combines a maximum entropy model, a Naive Bayes classifier, and a support vector machine. Statistical tools are employed to enhance the choice of material to be summarized. The dimension of the text, phrases of key value, text occurrence score throughout the entire document, the initializers, sentence relative position, and the frequency of less important information are among the features used to generate an effective summary. Other features include the similarity of words between sentences and paragraphs. The three parts of the approach developed by Zhong et al. [8] are concept extraction, summary creation, and reconstruction validation. The deep learning technique is used to minimize the information loss using the validation technique. Vital information in the paper is also identified using the dynamic programming approach. The algorithm outperforms many supervised algorithms in terms of multi-document summary generation. The statistical method to perform the word count to create the sentence vector was developed by Yao et al. [9]. The sentence vector is input to the hidden layer to compute a cluster of similar score words. The K-nearest neighbor method is used to identify similar words, which are used in text summarization. The proposed technique uses a deep neural network with a three-layer model.

Singh et al. [10] employed deep learning for bilingual (Hindi and English) unsupervised automated text summarization. RBMs, or restricted Boltzmann machines, are used to increase precision. An RBM possesses two secret layers. The pre-processed input is transmitted to layer one. Weights that are produced at random are multiplied by the input at layer one. To add up the input sentences, a bias value that is created at random is employed. For hidden layer two, an identical process is carried out. Extractive query-oriented single document summarization was proposed by Yousefi et al. [11], generating feature space from the term frequency (tf) input using a deep auto encoder (AE). Heena et al. [12] employed a hybrid model combining a fuzzy logic system and a deep neural network to summarize text. The model used four features for text summarization: title similarity, term weight, named entities, and numerical data. Based on the type of data, numerical or word-based named entities will be assigned. Finally, more weight sentences are extracted to produce an effective summary. Backpropagation neural networks (BNNs) perform better than existing models. To choose key phrases for text summarization, Nikhil et al. [13] used fuzzy logic with a restricted Boltzmann machine (RBM). The sentence position, sentence length, numerical token, and the frequency of sentences are the characteristics that are employed. The highest frequency sentence is used to divide each sentence into its component parts, and then each sentence is given a score.

Sahaba et al.’s configurable fuzzy features and neural sequence-to-sequence model with attention mechanism for word distribution in vocabulary and context are provided in [14]. Title, sentence proper size, weight matrix of sentences, proper noun, and alpha number data are employed as characteristics for fuzzy parts. Using a sequence-to-sequence approach, abstractive text summarization is accomplished. An LSTM-CNN-based ATS framework (ATSDL) has been developed by Song et al. [15] that can create new sentences by investigating semantic phrases, which are more fine-grained pieces than sentences. The three processes that make up phrase extraction are phrase acquisition, phrase improvement, and phrase combination. Wordnet is used to combine phrases. The text summarizing approach is put into practice using the LSTM-CNN model. The sentence similarity strategy for text summarization was developed by Abujar et al. [16]. The suggested approach is utilized mostly for Bengali and English. Word-to-word, sentence-to-sentence, and order vectors are utilized in lexical layer analysis to produce sentence similarity scores. The sequence-to-sequence approach was used by Al-maleh et al. [17] to summarize Arabic literature. On the basis of the original text, an abstractive headline is generated using an encoder–decoder structure. The context vector is used to identify terms that are not in dictionaries. A technique for summarising Arabic text that uses a sequence-to-sequence model was proposed by Wazery et al. [18]. Encoder and decoder are the two parts of this paradigm that function. The sequence-to-sequence active modelling technique verifies different criteria’s to see which one performs best.

3. Competitive Analysis

The comparative comparison of several enhanced machine learning methods with neural models for text summarization of text documents is presented in

Table 2. Comparative analysis of different deep learning methods for text summarization of text document/s.

Ref. No.	Technique Used	Pros	Cons	Corpus Used	Accuracy
[3]	Three-layer feed-forward network	Used for Vietnamese Text. No other work on Vietnamese text summarization	Human-generated summary is the comparison and evaluation required	Manual Corpus is generated for training and testing; 16,117 Sentences; 300 documents	For 80% text, it has 0.875 precision
[4]	Opposition-based learning	Enhance the DE evolutionary search algorithm used for better text clustering	RNN outperforms the method with respect to ROUGE score level 1	DUC 2002 dataset	ROUGE-2 score 0.22356
[5]	Statistical method and semantic method	Used for Malayalam text summarization Syntactic structure is used for effective summary generation	Time complexity is the key factor while large text summary generation	Manual Corpus is generated using 25 documents containing Malayalam text	ROUGE-L score precision–statistical method 0.637 semantic method 0.466
[6]	REGSUM model	Effective in word weight identification using a novel model to cluster high weight words	DPP outperforms the method with respect to ROUGE2 Score	DUC 2003 Dataset is utilized.	ROUGE-2 score 9.75
[7]	Hybrid model	Effective feature extraction using hybrid approach	CNN outperforms the method with multi later model	DUC 2002 used for analysis.	ROUGE-1 score 0.3862
[8]	Deep learning architecture	Query-based multi-document summarization is developed for English modelling	Ranking SVM outperforms in ROUGE 1,2 Score	DUC 2005, 2006, and 2007 datasets	ROUGE-SU-4 Score 0.1685
[9]	Deep neural network	Outperforms random, LEAD, and LSA algorithms in terms of ROUGE-L Score	DSDR-nonlinear Outperform s in ROUGE-1,2 and L Score	DUC 2006, 2007 Datasets used for training and testing purposes	ROUGE-L score 0.31008, 0.36881
[10]	Restricted Boltzmann machine	Used for bilingual text summarization	RNN outperforms the method in ROUGE-1 Score	Manual Corpus is created and used for testing purposes	ROUGE-1 score 0.85233
[11]	Ensemble noisy AutoEncoder techniques	Summarization using auto encode is generated	Time complexity issue increases with number of connected layers	SKE, BC3 Datasets used for training and testing purpose	ROUGE-2 score 0.5031
[12]	Hybrid model	Performs 31% better than individual models of fuzzy system and ANN	BNN outperform the methods using improved ROUGE-1 Score	Manual Corpus is generated for training and testing	ROUGE-1,2, L score for the model is 0.75
[13]	Hybrid model	Outperforms RBM method in ROUGE-1	CNN Outperform the Method in ROUGE-2 Score	Manual Corpus is generated for training and testing	ROUGE-1 score 0.84
[14]	Combined model	Outperforms sequence-to-sequence model using ROUGE-1 score	Feature customization leads to complex vocabulary count	CNN, Daily Mail Dataset	ROUGE-1 score 0.3619
[15]	LSTM-CNN model	Improved semantics and syntactic structure for word formation	Time complexity issue with increased layers	CNN, Daily Mail Corpus is used experiment	ROUGE-2 score 0.178
[16]	Sentence similarity measuring model	Used for Bengali text summarization using vocabulary-focused model	Backtracking methods outperform in ROUGE-L Score	Manual Corpus is generated for training and testing	Wu and Palmer measure (WP) 1
[17]	Sequence-to-sequence model for deep learning	Used for Arabic text summarization; structure-based approach is used	Transformer outperforms the method in ROGUE-2 score	Arabic dataset is generated for training and testing	ROUGE-1 score 0.4423
[18]	BiLSTM Model	Used for Arabic Text summarizations using syntax	Time complexity issue	Arabic dataset is generated for training and testing.	ROUGE-L Score 0.3437

[3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]. Analysis of several methods based on the deep learning method employed, benefits, drawbacks, dataset utilized, and accuracy in terms of ROUGE score. A set of criteria and software tool called Recall-Oriented Understudy for Gisting Evaluation (ROUGE) Score are used to assess automated summarization.

4. Conclusions

In this manuscript, we have explored various deep learning approaches for text summarization, based on experimental results proposed by various researchers. We conclude that every deep learning method has some pros and cons. Evaluating these methods, we found that the pre-trained transformer yields the best results for text summarization. In future work, combinations of traditional methods and pre-trained transformers will be investigated for better results. With the introduction of deep learning methods, the field of text summarization has made significant strides forward. However, a vast number of uncharted territories still need exploration. One of the main focuses is on developing new architectures that can pick up on the subtle meanings hidden within text. Multi-modal summarizing, which includes not just text but also images and maybe audio data, has the potential to significantly improve the summary creation process.

The breadth and variety of the dataset is a major cause for concern. It is possible that current datasets lack the diversity and size needed to effectively train models to understand nuanced or domain-specific settings. Therefore, expanding and diversifying existing datasets, perhaps by tapping into real-world data, is crucial for making progress in the field.

Incorporating transfer learning and few-shot learning has the ability to improve the summary quality while also reducing the data demand problem. It is also important to incorporate human-centric evaluations or to develop new metrics that better capture semantic coherence and informativeness when evaluating the performance of summarization models beyond traditional measures such as ROUGE.