Predicting Future Promising Technologies Using LSTM

Abstract

With advances in science and technology and changes in industry, research on promising future technologies has emerged as important. Furthermore, with the advent of a ubiquitous and smart environment, governments and enterprises are required to predict future promising technologies on which new important core technologies will be developed. Therefore, this study aimed to establish science and technology development strategies and support business activities by predicting future promising technologies using big data and deep learning models. The names of the “TOP 10 Emerging Technologies” from 2018 to 2021 selected by the World Economic Forum were used as keywords. Next, patents collected from the United States Patent and Trademark Office and the Science Citation Index (SCI) papers collected from the Web of Science database were analyzed using a time-series forecast. For each technology, the number of patents and SCI papers in 2022, 2023 and 2024 were predicted using the long short-term memory model with the number of patents and SCI papers from 1980 to 2021 as input data. Promising technologies are determined based on the predicted number of patents and SCI papers for the next three years. Keywords characterizing future promising technologies are extracted by analyzing abstracts of patent data collected for each technology and the term frequency-inverse document frequency is measured for each patent abstract. The research results can help business managers make optimal decisions in the present situation and provide researchers with an understanding of the direction of technology development.

1. Introduction

Owing to the development of science and technology and changes in industry, research on promising future technologies has become important. With the advent of a ubiquitous and smart environment, governments and enterprises are required to predict future promising technologies on which new important and core technologies will be developed [1]. Technology forecasting focuses on supporting business managers to make optimal decisions in the present situation and helps researchers understand the direction of technology development by predicting the future in detail through quantitative techniques. Through technological forecasting, it is possible to effectively link science and technology with economic development by forecasting the future situation and integrating economic needs and research opportunities. Technology forecasting is becoming an important tool to support the forecasting of industry and technological development [2]. As globalization accelerates and the industrial paradigm changes rapidly, technology forecasting for rapidly changing important technologies has emerged in response to the needs of the private and public sectors [3,4]. With quantitative analysis techniques being applied to technical forecasting [5,6], the reliability and validity of technical forecasting using papers and patents are increasing [7]. As patents and papers are representative data of technical information, these attempts of forecasting help solve the problem of subjective bias of experts [8,9]. In recent years, scholars have begun predicting technology based on the number of publications of papers and patents [10]. The common methods of technology forecasting through analysis of the number of papers and patents are regression, machine learning, and deep learning [11]. Deep learning has a more in-depth network structure than machine learning, which can significantly improve the prediction accuracy for the problems that require complex solutions [12]. Mudassir et al. [13] used a long short-term memory (LSTM) network for forecasting bitcoin price fluctuations. Cai et al. [14] forecasted wind power using a generalized regression neural network (GRNN) and showed that the deep learning approach has higher prediction accuracy. Gui and Xu [15] used a deep learning text classification model to extract relevant Science Citation Index (SCI) papers from the Web of Science database for the period 1996–2019 for topic classification and used Ensemble Empirical Mode Decomposition (EEMD) and LSTM neural networks to predict the future development of each research field. Zhou et al. [16] used a deep learning algorithm to predict emerging technologies in Gartner’s hype curve in 2017 based on patent data from 2000 to 2016. Lee et al. [17] devised a deep learning model based on meta-knowledge (i.e., text information including citations, abstracts, and area codes) for prediction of future growth potential.

This study aimed to establish science and technology development strategies and support business activities by predicting future promising technologies using big data and deep learning models. Herein, the promising technology names of the “TOP 10 Emerging Technologies” from 2018 to 2021 selected by the World Economic Forum (WEF) are used as keywords to analyze the patents collected from the United States Patent and Trademark Office (USPTO) and SCI papers collected from the Web of Science database by time-series forecast (TSF). For each technology, the number of patents and SCI papers in 2022, 2023 and 2024 are predicted using the LSTM model with the number of patents and SCI papers collected from 1980 to 2021 as input data. Promising technologies are determined based on the number of predicted patents and SCI papers for the next three years. This study has differences in previous works, in that big data are collected from the vast databases of the USPTO and the Web of Science and future promising technologies are derived based on it using a deep learning model. Furthermore, this study aimed to extract the keywords characterizing future promising technologies—this is achieved by calculating the term frequency-inverse document frequency (TF-IDF) of each word in a patent abstract by using the abstracts of patent data collected for each technology from the USPTO to compose the corpus.

2. Data and Methods

2.1. Data

In this study, the input dataset of LSTM was constructed using the number of patents collected from the USPTO registered between 1980 and 2021 for each technology and the number of SCI papers collected from the Web of Science database by keyword search for the promising technology names of the “TOP 10 Emerging Technologies” from 2018 to 2021 selected by the WEF, as shown in

Table 1. World Economic Forum “Top 10 Emerging Technologies” (2018–2021).

No.	2018	2019	2020	2021
1	Augmented reality	Bioplastics for a circular economy	Microneedles for painless injections and tests	Decarbonization rises
2	Personalized medicine	Social robots	Sun-powered chemistry	Crops that self-fertilize
3	AI-led molecular design	Lenses for miniature devices	Virtual patients	Breath sensors diagnose disease
4	More capable digital helpers	Disordered proteins as drug targets	Spatial computing	On-demand drug manufacturing
5	Implantable drug-making cells	Smarter fertilizers can reduce environmental contamination	Digital medicine	Energy from wireless signals
6	Gene drive	Collaborative telepresence	Electric aviation	Engineering better ageing
7	Algorithm for quantum computers	Advanced food tracking and packing	Lower-carbon cement	Green ammonia
8	Plasmonic materials	Safer nuclear reactors	Quantum sensing	Biomarker devices go wireless
9	Lab-grown meat	DNA data for storage	Green hydrogen	Houses printed with local materials
10	Electroceuticals	Utility-scale storage of renewable energy	Whole-genome synthesis	Space connects the globe

.

Table 2. Number of patents and papers collected for technology in Table 1 (the left shows the number of patents and the right shows the number of papers).

No.	2018	2019	2020	2021	Sum
1	41,088/16,204	0/234	0/5	0/25
2	134/745	91/4267	0/1	0/3
3	0/14	0/31	91/8923	0/16
4	0/51	0/11	209/46,898	0/13
5	0/5	0/124	49/16,237	0/0
6	152/737	6/154	7/1594	0/0
7	2/7	0/6	0/32	12/455
8	360/1735	1/980	279/4732	0/0
9	0/0	18/128	40/3314	0/0
10	27/14	0/36	5/27	0/0
Sum	41,763/19,512	116/5971	680/81,763	12/512	42,571/107,758

shows the total number of patents collected from the USPTO between 1980 and 2021 for each technology and the total number of SCI papers collected from the Web of Science database for technology in Table 1. The input dataset of the LSTM model was created by composing the number of patents and SCI papers collected for technology and year as follows. To predict the number of patents for the next three years for each technology, vector $\left(\left(\mathrm{t}, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}}\right), \left(\mathrm{t}+1, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+1}\right),\cdots , \left(\mathrm{t}+9, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+9}\right) \right)$ with length $10$ was constructed for year $\mathrm{t} \left(\mathrm{t}=1980, 1981, \cdots , 2012\right)$.

To predict the number of SCI papers for the next three years for each technology, vector $\left(\left(\mathrm{t}, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}}\right), \left(\mathrm{t}+1, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+1}\right),\cdots , \left(\mathrm{t}+9, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+9}\right) \right)$ with length $10$ was constructed for year $\mathrm{t} \left(\mathrm{t}=1980, 1981, \cdots , 2012\right)$. The LSTM model was modeled to predict the number of patents for the next three years $\left(\left(\mathrm{t}+10, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+10}\right), \left(\mathrm{t}+11, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+11}\right), \left(\mathrm{t}+12, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+12}\right) \right)$ for input data $\left(\left(\mathrm{t}, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}}\right), \left(\mathrm{t}+1, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+1}\right),\cdots , \left(\mathrm{t}+9, \mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}+9}\right) \right)$ and the number of SCI papers for the next three years $\left(\left(\mathrm{t}+10, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+10}\right), \left(\mathrm{t}+11, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+11}\right), \left(\mathrm{t}+12, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+12}\right) \right)$ for input data $\left(\left(\mathrm{t}, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}}\right), \left(\mathrm{t}+1, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+1}\right),\cdots , \left(\mathrm{t}+9, \mathrm{paper}\_{\mathrm{num}}_{\mathrm{t}+9}\right) \right)$. An increase in the number of patents and SCI papers predicted for the next three years compared to that of 2021 indicated a promising technology in the future. To extract keywords that characterize future promising technologies, the abstract of each patent was considered one document for each technology and the set of abstracts as a corpus through data mining to calculate the term frequency (TF), document frequency (DF), and term frequency-inverse document frequency (TF-IDF). Keywords characterizing future promising technologies were extracted from words with a calculated TF-IDF.

2.2. Model

A recurrent neural network (RNN) is a deep learning model that uses time-series data from the past as input and outputs future data; for example, a river level prediction model [18,19], solar power generation prediction model [20,21], fine dust prediction model [22,23], energy demand prediction model [24], and stock price prediction model [25,26]. In this study, using the dataset in Section 2.1 as input data, the predictions made by LSTM for promising future technologies showed excellent performance even with a dataset having long-term dependencies.

The mathematical model of the LSTM is expressed as Equation (1) and illustrated in Figure 1. The output $h_t$, output gate $o_t$, new memory content $\widetilde{c_t}$, forget gate $f_t$, and input gate $i_t$ of the LSTM are expressed as Equation (1).

Output	$h_t=o_{t }tanh\left(c_t\right)$
Output gate	$o_t=\sigma \left(W_o{x}_t+U_o{h}_{t-1}+V_o{c}_t\right)$
Memory cell	$c_t=f_t{c}_{t-1}+i_t{\tilde{c}}_t$
New memory content	${\tilde{c}}_t=tanh\left(W_c{x}_t+U_c{h}_{t-1}\right)$	(1)
Forget gate	$f_t=\sigma \left(W_f{x}_t+U_f{h}_{t-1}+V_f{c}_{t-1}\right)$
Input gate	$i_t=\sigma \left(W_i{x}_t+U_i{h}_{t-1}+V_i{c}_{t-1}\right)$

Equation (1) can be illustrated as an image in Figure 1.

To improve the prediction accuracy of LSTM, a stacked LSTM, as shown in Figure 2, was used as a model by stacking two LSTM layers with a hidden size of 100. The experiment was configured as follows, and it was confirmed that the predicted values converged when the epochs were set to 200.

Epochs: 200, Hidden size: 100, Loss function: MSE, Optimizer: SGD, Learning rate: 0.001.

2.3. Patent Analysis Results

Technologies with less than three patents in the patent dataset were excluded from the analysis due to difficulties in constructing a sufficient training dataset for the LSTM.

Table 3. Rate of increase and prediction accuracy of the predicted number of patents by technology.

No.	Technology	Rate of Increase	Accuracy (%)
1	Augmented reality	106.8433	78.62
2	Collaborative telepresence	0.049675	73.44
3	Digital medicine	0.2259	94.50
4	DNA data storage	0.00825	96.85
5	Electric aviation	0.037975	82.18
6	Electroceuticals	0.40255	93.50
7	Gene drive	0.092075	94.76
8	Green ammonia	−0.00878	79.60
9	Green hydrogen	0.22745	90.70
10	Personalized medicine	0.447675	94.41
11	Plasmonic materials	2.45425	95.53
12	Quantum sensing	1.8588	97.55
13	Social robots	0.82	92.17
14	Spatial computing	2.295025	94.50
15	Virtual patients	2.4056	92.62
16	Whole-genome synthesis	0.0077	89.36

shows the results of calculating the rate of increase $(\mathrm{patent}\_{\mathrm{num}}_{2024}-\mathrm{patent}\_{\mathrm{num}}_{2021})/4$ in the number of patents in 2024 compared to 2021 based on the number of patents expected to be applied in the next three years using the LSTM model for 16 technologies. In Table 3, the accuracy was calculated as follows.

With the maximum number of patents $\mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}}$ collected under each technology for year $\mathrm{t} \left(\mathrm{t}=1980, 1981, \cdots , 2012\right)$ as max, the number of patents, which was the input data of the LSTM, and the number of patents, the output data, were multiplied by $100/\max$ to normalize the number of patents to a number from 0 to 100. Let ${\mathrm{difference}}_{\mathrm{t}}$ be the absolute value of the difference between normalized $\mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}}$, which represents input data, and normalized $\mathrm{patent}\_{\mathrm{num}}_{\mathrm{t}}$, which represents output data in the same year. Prediction accuracy was calculated as $\mathrm{accuracy}=100-\mathrm{mean}\left({\mathrm{difference}}_{\mathrm{t}}\right)$.

Based on the predicted increase in the number of patents over the next three years, “Augmented reality” was predicted to be the most promising technology in the future, followed by “Plasmonic materials,” “Virtual patients,” “Spatial computing,” “Quantum sensing,” “Social robots,” “Personalised medicine,” $\cdots$. Figure 3 shows the input data of future promising technologies and the predicted number of patents as a graph. Since the model used in this study is a predictive model based on past data, it tends to underestimate when a sudden increase in a short period of time is observed in the input data.

The number of patents in the next three years predicted by the LSTM model using the number of patents of the past 10 years as input data showed a tendency to conservatively predict the number of patents with a smaller variation from the number of patents actually observed.

In this study, the abstracts of patent data collected by technology were used to compose the corpus to calculate the TF of each abstract word appearing in the corpus; the DF, the number of documents in which each abstract word appeared; and the TF-IDF [28], a statistical number indicating how important each abstract word was in the corpus, and to extract the keywords characterizing future promising technologies.

Table 4. Keywords extracted based on TF-IDF.

(a) Keywords of Plasmonic Materials
Word	TF	DF	TF_IDF
Layer	355	103	442.77
Material	353	124	375.36
Surface	326	130	331.36
Structure	158	49	312.78
Peg	133	36	303.34
Light	194	78	295.31
Optical	158	59	283.97
Region	108	26	280.35
Waveguide	126	50	246.94
Plasmonic	206	111	241.67
Metal	105	39	231.29
Devices	166	91	227.40
Transducer	186	106	226.70
Least	179	102	224.99
Positioned	109	46	222.52
Magnetic	105	43	221.28
Portion	114	51	221.21
Substrate	150	84	217.35
Nft	109	50	213.62
Dielectric	102	44	212.67
Nearfield	100	48	199.98
Oxide	69	21	193.24
Configured	105	59	188.72
Device	115	74	181.03
Field	131	90	180.88
Film	70	29	174.33
Thereof	73	33	172.67
Conductive	70	30	172.04
Electromagnetic	66	26	171.32
(b) Keywords of Quantum Sensing
Word	TF	DF	TF_IDF
Layer	203	45	369.53
Material	285	93	315.12
Light	181	54	297.14
Quantum	152	42	286.94
Diamond	150	41	286.70
Optical	180	59	279.83
Magnetic	171	67	244.44
Spin	80	13	240.79
Field	122	47	216.89
Excitation	97	32	208.79
Device	183	90	208.27
Semiconductor	132	58	207.43
Nanometers	48	4	193.90
Configured	124	59	192.77
Signal	102	42	192.55
Region	72	20	187.52
Substrate	105	49	182.38
Source	98	44	180.55
Surface	90	41	172.02
Frequency	83	39	162.69
Defect	61	19	161.85
Diode	55	14	161.75
Detector	97	56	155.77
System	87	53	144.42
Sensor	61	26	143.54
Magnetooptical	46	12	141.87
Unit	41	8	141.52
Micro	44	11	139.22
Array	57	25	136.28

TF, term frequency; DF, document frequency; TF_IDF, term frequency-inverse document frequency.

shows the results of extracting the keywords of future promising technologies based on the TF-IDF for each technology. Keywords for technologies are not shown in Table 4 but are included in the Supplementary Data in the online resource.

2.4. Results of SCI Paper Analysis

Among the data of the SCI papers in Section 2.1, technologies with less than four published papers were excluded from the analysis due to difficulties in constructing a sufficient training dataset for LSTM.

Table 5. Rate of increase and prediction accuracy of the predicted number of Science Citation Index (SCI) papers by technology.

No.	Technology	Rate of Increase	Accuracy (%)
1	Advanced food tracking and packaging	0.039575	84.32
2	AI-led molecular design	0.022825	91.39
3	Algorithms for quantum computers	0.013575	82.32
4	Augmented reality	78.9371	81.91
5	Bioplastics for a circular economy	2.066675	94.70
6	Breath sensors diagnose disease	0.015575	74.27
7	Collaborative telepresence	−0.23878	92.32
8	Decarbonization rises	−0.04815	88.86
9	Digital medicine	39.60455	85.55
10	Disordered proteins as drug targets	0.003725	76.20
11	DNA data for storage	−0.31628	91.49
12	Electric aviation	3.43585	76.81
13	Electroceuticals	0.094375	90.43
14	Gene drive	0.116	81.29
15	Green ammonia	1.25715	94.31
16	Green hydrogen	2.0026	92.00
17	Implantable drug-making cells	0.0007	73.12
18	Lower-carbon cement	0.158525	84.36
19	Microneedles for painless injections and tests	0.0023	73.42
20	More capable digital helpers	0.33175	89.39
21	On-demand drug manufacturing	0.0144	88.04
22	Personalized medicine	1.216357	88.90
23	Plasmonic materials	6.9311	84.02
24	Quantum sensing	5.735575	71.52
25	Safer nuclear reactors	−0.0415	84.85
26	Smarter fertilizers can reduce environmental contamination	−0.16527	95.13
27	Social robots	9.46255	76.76
28	Spatial computing	71.04438	70.20
29	Tiny lenses for miniature devices	0.097625	75.01
30	Utility-scale storage of renewable energy	−0.29128	93.85
31	Virtual patients	34.59743	93.26
32	Whole-genome synthesis	0.053025	87.81

shows the results of calculating the rate of increase $(\mathrm{paper}\_{\mathrm{num}}_{2024}-\mathrm{paper}\_{\mathrm{num}}_{2021})/4$ in the number of papers in 2024 compared to 2021 based on the number of papers predicted to be published in the next three years using the LSTM model for 32 technologies. Table 5 shows the accuracy calculations, similar to the calculations presented in the patent analysis results.

Based on the predicted increase in the number of SCI papers over the next three years, the most promising technology was predicted to be “Augmented reality,” followed by “Spatial computing,” “Digital medicine,” “Virtual patients,” “Social robots,” “Plasmonic materials,” “Quantum sensing,” $\cdots$. Figure 4 shows the input data of future promising technologies and the predicted number of SCI papers. As in Figure 3, when a sudden increase in a short period of time is observed in the input data, the predicted value tends to be underestimated.

As shown in

Table 6. Technologies predicted to grow in both number of patents and Science Citation Index (SCI) papers.

Technology	Rate of Increase (SCI Papers)	Rate of Increase (Patents)
Augmented reality	78.9371	106.8433
Spatial computing	71.04438	2.295025
Digital medicine	39.60455	0.2259
Virtual patients	34.59743	2.4056
Social robots	9.46255	0.82
Plasmonic materials	6.9311	2.45425
Quantum sensing	5.735575	1.8588
Electric aviation	3.43585	0.037975
Green hydrogen	2.0026	0.22745
Personalized medicine	1.216357	0.447675
Gene drive	0.116	0.092075
Electroceuticals	0.094375	0.40255
Whole-genome synthesis	0.053025	0.0077

, the technologies predicted by the LSTM to grow in both number of patents and SCI papers in the next three years included: “Augmented reality,” “Spatial computing,” “Digital medicine,” “Virtual patients,” “Social robots,” “Plasmonic materials,” “Quantum sensing,” “Electric aviation,” “Green hydrogen,” “Personalized medicine,” “Gene drive,” “Electroceuticals,” and “Whole-genome synthesis.” Figure 5 shows a graph of the predicted number of patents and SCI papers for the top 10 technologies with a high growth rate among these 13 technologies.

3. Conclusions

This study used the promising technology names of the “TOP 10 Emerging Technologies” from 2018 to 2021 selected by the WEF as keywords to analyze the patents collected from the USPTO and SCI papers collected from the Web of Science database by TSF. Using the number of patents and SCI papers collected for 40 technologies as input data, the number of patents and SCI papers in the next three years was predicted for each technology using a two-layer LSTM model. Promising technologies were derived based on the increase rate of the predicted number of patents and the increase rate of the predicted number of SCI papers. This study is meaningful in that it determines promising technologies with an average accuracy of 86.42% using a deep learning model for two databases for 40 broad technologies.

The 13 technologies predicted to grow in both the number of patents and the number of SCI papers in the next three years, namely, “Augmented reality,” “Spatial computing,” “Digital medicine,” “Virtual patients,” “Social robots,” “Plasmonic materials,” “Quantum sensing,” “Electric aviation,” “Green hydrogen,” “Personalized medicine,” “Gene drive,” “Electroceuticals,” and “Whole-genome synthesis,” can be considered future promising technologies. Using the research results, business managers can make optimal decisions in the present situation and researchers can understand the direction of technology development. Through technological forecasting, it is possible to effectively link science and technology with economic development by forecasting the future situation more similarly and integrating economic needs and research opportunities.

Furthermore, this study differs from other studies in that keywords characterizing future promising technologies were extracted by calculating the TF-IDF of each word in a patent abstract by using the abstracts of patent data collected for each technology from the USPTO to compose the corpus.

In the future, to determine promising technologies, a wider database will be built and other models that can further improve prediction accuracy will be investigated.