A Question and Answering Service of Typhoon Disasters Based on the T5 Large Language Model
Abstract
:1. Introduction
2. Related Works
3. Method
3.1. T5 Model Theory
3.2. Model Enhancement
3.3. Model Evaluation
- (1)
- ROUGE-N (N = 1, 2)
- (2)
- ROUGE-L
4. Experimental Results and Analysis
4.1. Experimental Environment
4.2. Experimental Procedure
4.3. Experimental Rusults
4.3.1. Model Metric Evaluation
4.3.2. Intelligent Evaluation
4.3.3. Manual Evaluation
4.3.4. Comparison between T5-Base and T5-Large
5. Discussion
5.1. Scalability
5.2. Exploration of Application Scenarios
5.3. Analysis of Limitation
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Knutson, T.R.; McBride, J.L.; Chan, J.; Emanuel, K.; Holland, G.; Landsea, C.; Held, I.; Kossin, J.P.; Srivastava, A.; Sugi, M. Tropical cyclones and climate change. Nat. Geosci. 2010, 3, 157–163. [Google Scholar] [CrossRef]
- Elsner, J.B.; Elsner, S.C.; Jagger, T.H. The increasing efficiency of tornado days in the United States. Clim. Dyn. 2015, 45, 651–659. [Google Scholar] [CrossRef]
- Murakami, H.; Wang, B. Patterns and frequency of projected future tropical cyclone genesis are governed by dynamic effects. Commun. Earth Environ. 2022, 3, 77. [Google Scholar] [CrossRef]
- Sarker, M.N.I.; Peng, Y.; Yiran, C.; Shouse, R.C. Disaster resilience through big data: Way to environmental sustainability. Int. J. Disaster. Risk Reduct. 2020, 51, 101769. [Google Scholar] [CrossRef]
- Yang, J.; Li, Y.; Liu, Q.; Li, L.; Feng, A.; Wang, T.; Zheng, S.; Xu, A.; Lyu, J. Brief introduction of medical database and data mining technology in big data era. J. Evid. Based Med. 2020, 13, 57–69. [Google Scholar] [CrossRef] [PubMed]
- Zhou, C.; Su, F.; Pei, T.; Zhang, A.; Du, Y.; Luo, B.; Cao, Z.; Wang, J.; Yuan, W.; Zhu, Y. COVID-19: Challenges to GIS with big data. Geogr. Sustain. 2020, 1, 77–87. [Google Scholar] [CrossRef]
- Naeem, M.; Jamal, T.; Diaz-Martinez, J.; Butt, S.A.; Montesano, N.; Tariq, M.I.; De-la-Hoz-Franco, E.; De-La-Hoz-Valdiris, E. Trends and future perspective challenges in big data. In Advances in Intelligent Data Analysis and Applications, Proceeding of the Sixth Euro-China Conference on Intelligent Data Analysis and Applications, Arad, Romania, 15–18 October 2019; Springer: Singapore, 2022; pp. 309–325. [Google Scholar]
- Liu, N.F.; Zhang, T.; Liang, P. Evaluating verifiability in generative search engines. arXiv 2023, arXiv:2304.09848. [Google Scholar]
- Shams, A.B.; Hoque Apu, E.; Rahman, A.; Sarker Raihan, M.M.; Siddika, N.; Preo, R.B.; Hussein, M.R.; Mostari, S.; Kabir, R. Web search engine misinformation notifier extension (SEMiNExt): A machine learning based approach during COVID-19 Pandemic. Healthcare 2021, 9, 156. [Google Scholar] [CrossRef] [PubMed]
- Zaib, M.; Zhang, W.E.; Sheng, Q.Z.; Mahmood, A.; Zhang, Y. Conversational question answering: A survey. Knowl. Inf. Syst. 2022, 64, 3151–3195. [Google Scholar] [CrossRef]
- Martinez-Gil, J. A survey on legal question–answering systems. Comput. Sci. Rev. 2023, 48, 100552. [Google Scholar] [CrossRef]
- Huang, D.; Wei, Z.; Yue, A.; Zhao, X.; Chen, Z.; Li, R.; Jiang, K.; Chang, B.; Zhang, Q.; Zhang, S. DSQA-LLM: Domain-Specific Intelligent Question Answering Based on Large Language Model. In Proceedings of the International Conference on AI-Generated Content, Shanghai, China, 25–26 August 2023; pp. 170–180. [Google Scholar]
- Kasneci, E.; Seßler, K.; Küchemann, S.; Bannert, M.; Dementieva, D.; Fischer, F.; Gasser, U.; Groh, G.; Günnemann, S.; Hüllermeier, E. ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 2023, 103, 102274. [Google Scholar] [CrossRef]
- Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Adv. Neural Inf. Process. Syst. 2024, 36, 11809–11822. [Google Scholar]
- Lyu, Y.; Li, Z.; Niu, S.; Xiong, F.; Tang, B.; Wang, W.; Wu, H.; Liu, H.; Xu, T.; Chen, E. CRUD-RAG: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv 2024, arXiv:2401.17043. [Google Scholar]
- Siriwardhana, S.; Weerasekera, R.; Wen, E.; Kaluarachchi, T.; Rana, R.; Nanayakkara, S. Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering. Trans. Assoc. Comput. Linguist. 2023, 11, 1–17. [Google Scholar] [CrossRef]
- Tang, Y.; Yang, Y. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv 2024, arXiv:2401.15391. [Google Scholar]
- Krause, A.; Cohen, S. Geographic Information Retrieval Using Wikipedia Articles. In Proceedings of the ACM Web Conference, Austin, TX, USA, 30 April–4 May 2023; pp. 3331–3341. [Google Scholar]
- Witmer, J.T. Mining Wikipedia for Geospatial Entities and Relationships. Doctoral Dissertation, University of Colorado at Colorado Springs, Colorado Springs, CO, USA, 2009. [Google Scholar]
- Choukolaei, H.A.; Ghasemi, P.; Goodarzian, F. Evaluating the efficiency of relief centers in disaster and epidemic conditions using multi-criteria decision-making methods and GIS: A case study. Int. J. Disaster Risk Reduct. 2023, 85, 103512. [Google Scholar] [CrossRef] [PubMed]
- Clemente-Suárez, V.J.; Navarro-Jiménez, E.; Ruisoto, P.; Dalamitros, A.A.; Beltran-Velasco, A.I.; Hormeño-Holgado, A.; Laborde-Cárdenas, C.C.; Tornero-Aguilera, J.F. Performance of fuzzy multi-criteria decision analysis of emergency system in COVID-19 pandemic. An extensive narrative review. Int. J. Environ. Res. Public Health 2021, 18, 5208. [Google Scholar] [CrossRef] [PubMed]
- Esmaelian, M.; Tavana, M.; Santos Arteaga, F.J.; Mohammadi, S. A multicriteria spatial decision support system for solving emergency service station location problems. Int. J. Geogr. Inf. Sci. 2015, 29, 1187–1213. [Google Scholar] [CrossRef]
- Saha, A.K.; Agrawal, S. Mapping and assessment of flood risk in Prayagraj district, India: A GIS and remote sensing study. Nanotechnol. Environ. Eng. 2020, 5, 11. [Google Scholar] [CrossRef]
- Yang, W.; Zhang, L.; Liang, C. Agricultural drought disaster risk assessment in Shandong Province, China. Nat. Hazards 2023, 118, 1515–1534. [Google Scholar] [CrossRef]
- Shao, Y.; Wang, Z.; Feng, Z.; Sun, L.; Yang, X.; Zheng, J.; Ma, T. Assessment of China’s forest fire occurrence with deep learning, geographic information and multisource data. J. For. Res. 2023, 34, 963–976. [Google Scholar] [CrossRef]
- Jena, R.; Pradhan, B.; Beydoun, G. Earthquake vulnerability assessment in Northern Sumatra province by using a multi-criteria decision-making model. Int. J. Disaster Risk Reduct. 2020, 46, 101518. [Google Scholar] [CrossRef]
- Fang, G.; Pang, W.; Zhao, L.; Cui, W.; Zhu, L.; Cao, S.; Ge, Y. Extreme typhoon wind speed mapping for coastal region of China: Geographically weighted regression–based circular subregion algorithm. J. Struct. Eng. 2021, 147, 04021146. [Google Scholar] [CrossRef]
- Wang, S.; Mu, L.; Yao, Z.; Gao, J.; Zhao, E.; Wang, L. Assessing and zoning of typhoon storm surge risk with a geographic information system (GIS) technique: A case study of the coastal area of Huizhou. Nat. Hazards Earth Syst. Sci. 2021, 21, 439–462. [Google Scholar] [CrossRef]
- Wu, K.; Wu, J.; Ding, W.; Tang, R. Extracting disaster information based on Sina Weibo in China: A case study of the 2019 Typhoon Lekima. Int. J. Disaster Risk Reduct. 2021, 60, 102304. [Google Scholar] [CrossRef]
- Zhang, T.; Cheng, C. Temporal and spatial evolution and influencing factors of public sentiment in natural disasters—A case study of typhoon haiyan. ISPRS Int. J. Geo-Inf. 2021, 10, 299. [Google Scholar] [CrossRef]
- Sufi, F.K.; Khalil, I. Automated disaster monitoring from social media posts using AI-based location intelligence and sentiment analysis. IEEE Trans. Comput. Soc. Syst. 2022, 1–11. [Google Scholar] [CrossRef]
- Rao, P.R.; Jhawar, T.N.; Kachave, Y.A.; Hirlekar, V. Generating QA from Rule-based Algorithms. In Proceedings of the 2022 International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India, 16–18 March 2022; pp. 1697–1703. [Google Scholar]
- Thorat, S.A.; Jadhav, V. A review on implementation issues of rule-based chatbot systems. In Proceedings of the International Conference on Innovative Computing & Communications (ICICC), Delhi, India, 21–23 February 2020. [Google Scholar]
- Jin, S.; Lian, X.; Jung, H.; Park, J.; Suh, J. Building a deep learning-based QA system from a CQA dataset. Decis. Support Syst. 2023, 175, 114038. [Google Scholar] [CrossRef]
- Abdel-Nabi, H.; Awajan, A.; Ali, M.Z. Deep learning-based question answering: A survey. Knowl. Inf. Syst. 2023, 65, 1399–1485. [Google Scholar] [CrossRef]
- Huang, X.; Zhang, J.; Li, D.; Li, P. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining, Melbourne VIC, Australia, 11–15 January 2019; pp. 105–113. [Google Scholar]
- Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A.H.; Riedel, S. Language models as knowledge bases? arXiv 2019, arXiv:1909.01066. [Google Scholar]
- Da, J.; Bras, R.L.; Lu, X.; Choi, Y.; Bosselut, A. Analyzing commonsense emergence in few-shot knowledge models. arXiv 2021, arXiv:2101.00297. [Google Scholar]
- Safavi, T.; Koutra, D. Relational world knowledge representation in contextual language models: A review. arXiv 2021, arXiv:2104.05837. [Google Scholar]
- Hu, Y.; Mai, G.; Cundy, C.; Choi, K.; Lao, N.; Liu, W.; Lakhanpal, G.; Zhou, R.Z.; Joseph, K. Geo-knowledge-guided GPT models improve the extraction of location descriptions from disaster-related social media messages. Int. J. Geogr. Inf. Sci. 2023, 37, 2289–2318. [Google Scholar] [CrossRef]
- Bhandari, P.; Anastasopoulos, A.; Pfoser, D. Are large language models geospatially knowledgeable? In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, Hamburg, Germany, 13–16 November 2023; pp. 1–4. [Google Scholar]
- Jiang, Z.; Araki, J.; Ding, H.; Neubig, G. How can we know when language models know? on the calibration of language models for question answering. Trans. Assoc. Comput. Linguist. 2021, 9, 962–977. [Google Scholar] [CrossRef]
- Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S.S.; Wei, J.; Chung, H.W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S. Large language models encode clinical knowledge. Nature 2023, 620, 172–180. [Google Scholar] [CrossRef] [PubMed]
- Thirunavukarasu, A.J.; Ting, D.S.J.; Elangovan, K.; Gutierrez, L.; Tan, T.F.; Ting, D.S.W. Large language models in medicine. Nat. Med. 2023, 29, 1930–1940. [Google Scholar] [CrossRef]
- Cui, J.; Li, Z.; Yan, Y.; Chen, B.; Yuan, L. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv 2023, arXiv:2306.16092. [Google Scholar]
- Wu, S.; Irsoy, O.; Lu, S.; Dabravolski, V.; Dredze, M.; Gehrmann, S.; Kambadur, P.; Rosenberg, D.; Mann, G. Bloomberggpt: A large language model for finance. arXiv 2023, arXiv:2303.17564. [Google Scholar]
- Yang, H.; Liu, X.-Y.; Wang, C.D. FinGPT: Open-Source Financial Large Language Models. arXiv 2023, arXiv:2306.06031. [Google Scholar] [CrossRef]
- Huang, J.; Wang, H.; Sun, Y.; Shi, Y.; Huang, Z.; Zhuo, A.; Feng, S. ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, 14–18 August 2022; pp. 3029–3039. [Google Scholar]
- Gao, Y.; Xiong, Y.; Wang, S.; Wang, H. GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest. Appl. Sci. 2022, 12, 12942. [Google Scholar] [CrossRef]
- Zhang, W.; Cai, M.; Zhang, T.; Zhuang, Y.; Mao, X. Earthgpt: A universal multi-modal large language model for multi-sensor image comprehension in remote sensing domain. arXiv 2024, arXiv:2401.16822. [Google Scholar]
- Guo, X.; Lao, J.; Dang, B.; Zhang, Y.; Yu, L.; Ru, L.; Zhong, L.; Huang, Z.; Wu, K.; Hu, D. Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. arXiv 2023, arXiv:2312.10115. [Google Scholar]
- Muhtar, D.; Li, Z.; Gu, F.; Zhang, X.; Xiao, P. LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model. arXiv 2024, arXiv:2402.02544. [Google Scholar]
- Ni, J.; Ábrego, G.H.; Constant, N.; Ma, J.; Hall, K.B.; Cer, D.; Yang, Y. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv 2021, arXiv:2108.08877. [Google Scholar]
- Karimzadeh, M.; Pezanowski, S.; MacEachren, A.M.; Wallgrün, J.O. GeoTxt: A scalable geoparsing system for unstructured text geolocation. Trans. GIS 2019, 23, 118–136. [Google Scholar] [CrossRef]
- Khattab, O.; Zaharia, M. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, China, 25–30 July 2020; pp. 39–48. [Google Scholar]
- Zhou, Y.; Li, C.; Huang, G.; Guo, Q.; Li, H.; Wei, X. A Short-Text Similarity Model Combining Semantic and Syntactic Information. Electronics 2023, 12, 3126. [Google Scholar] [CrossRef]
- Bag, S.; Kumar, S.K.; Tiwari, M.K. An efficient recommendation generation using relevant Jaccard similarity. Inf. Sci. 2019, 483, 53–64. [Google Scholar] [CrossRef]
- Verma, V.; Aggarwal, R.K. A comparative analysis of similarity measures akin to the Jaccard index in collaborative recommendations: Empirical and theoretical perspective. Soc. Netw. Anal. Min. 2020, 10, 43. [Google Scholar] [CrossRef]
- Lin, C.Y. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out; Association for Computational Linguistics: Barcelona, Spain, 2004; pp. 74–81. [Google Scholar]
- Shazeer, N.; Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018; pp. 4596–4604. [Google Scholar]
- Zheng, L.; Chiang, W.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv 2023, arXiv:2306.05685. [Google Scholar]
- Wang, C.; Cheng, S.; Xu, Z.; Ding, B.; Wang, Y.; Zhang, Y. Evaluating open question answering evaluation. arXiv 2023, arXiv:2305.12421. [Google Scholar]
Type of Data | Data Description | Demonstration | |
---|---|---|---|
Typhoon Meteorological Data | From a meteorological perspective, describe fundamental knowledge related to typhoon concepts. | Typhoon Definition | A typhoon is a tropical cyclone that develops between 180° and 100° E in the Northern Hemisphere. |
Typhoon Naming | Since 2000, the tropical cyclone naming list in the northwest Pacific has been developed by the WMO Typhoon Committee. There are five naming lists, each consisting of two names provided by 14 members. | ||
Typhoon Classification | A tropical depression is upgraded to a tropical storm should its sustained wind speeds exceed 34 knots. Should the storm intensify further and reach sustained wind speeds of 48 knots then it will be classified as a severe tropical storm. | ||
...... | |||
Typhoon Disaster Case Data | From a disaster studies perspective, select relevant information about Typhoon “In-Fa” from historical occurrences of typhoons, and describe the disasters it caused and their associated impacts. | Evolution Mechanism | “In-Fa” has a structurally complete and symmetrical form, with a clear eye of the typhoon and a vast expanse of cloud cover. True to its name, it is a “beautiful” typhoon. |
Characteristics and Attributes | On 25 July, the Typhoon “In-Fa” made landfall along the coast of Putuo District, Zhoushan City, Zhejiang Province, around 12:30 p.m. The maximum wind force near the center reached 13 on the Beaufort scale (38 m per second), with the minimum central pressure of 965 hPa. | ||
Disaster Situation Information | Before making landfall, Typhoon “In-Fa” had already impacted the climate on the Chinese mainland. On 20 July, Henan Province experienced catastrophic extreme precipitation, which results in the deaths of 302 people. | ||
...... | |||
Typhoon Disaster Management Data | From a disaster management perspective, describe the relevant knowledge generated by humans to prevent and mitigate typhoon disasters. | Typhoon Forecast and Warning | Typhoon warnings are issued by specialized agencies in various regions during the period when the storm may strike, providing forecasts and alerts. |
Emergency Response Measures | When a typhoon approaches, it is necessary to secure doors and windows tightly, remove all kinds of hanging objects indoors and outdoors, close doors and windows, and if necessary, reinforce them with nailed wooden boards. | ||
Disaster Recovery | All levels of government departments mobilize the entire population and achieve full coverage, carrying out emergency rescue, garbage cleaning, sludge and pollution removal, and disinfection and sterilization, to ensure no major epidemic outbreaks after major disasters. | ||
...... |
Model | T5-Large no_Context | T5-Large with_Context | Typhoon-T5-Large no_Context | Typhoon-T5-Large with_Context |
---|---|---|---|---|
ROUGE-1 | 11.36% | 25.45% | 30.45% | 40.82% |
ROUGE-2 | 6.23% | 10.57% | 12.15% | 21.72% |
ROUGE-L | 16.81% | 26.36% | 29.09% | 37.27% |
Len(Min_Length) | Question | Answer |
---|---|---|
128 | What is the initial stage of typhoon formation? | A tropical depression. At this stage, the initial low-pressure circulation needs to evolve until forming a complete typhoon structure. |
128 | How are medical services and health facilities guaranteed during emergency response? | Emergency response methods for ensuring medical services and health facilities include preparing drugs and medical equipment in advance, dispatching medical personnel, setting up emergency medical stations, etc., to deal with possible public health problems. |
64 | How are medical services and health facilities guaranteed during emergency response? | Preparations involve securing drugs and medical equipment, dispatching personnel, and establishing emergency medical stations. |
64 | What is the initial stage of typhoon formation? | A tropical depression is the initial stage of typhoon formation |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Xia, Y.; Huang, Y.; Qiu, Q.; Zhang, X.; Miao, L.; Chen, Y. A Question and Answering Service of Typhoon Disasters Based on the T5 Large Language Model. ISPRS Int. J. Geo-Inf. 2024, 13, 165. https://doi.org/10.3390/ijgi13050165
Xia Y, Huang Y, Qiu Q, Zhang X, Miao L, Chen Y. A Question and Answering Service of Typhoon Disasters Based on the T5 Large Language Model. ISPRS International Journal of Geo-Information. 2024; 13(5):165. https://doi.org/10.3390/ijgi13050165
Chicago/Turabian StyleXia, Yongqi, Yi Huang, Qianqian Qiu, Xueying Zhang, Lizhi Miao, and Yixiang Chen. 2024. "A Question and Answering Service of Typhoon Disasters Based on the T5 Large Language Model" ISPRS International Journal of Geo-Information 13, no. 5: 165. https://doi.org/10.3390/ijgi13050165