Beyond the List: A Framework for the Design of Next-Generation MEDLINE Search Tools
Abstract
1. Introduction
2. Background
Traditional Search Tools
3. A Framework for the Design of MEDLINE Search Tools
3.1. Visualization
3.1.1. The Problem: Textual Overload in MEDLINE Search
3.1.2. The Solution: Applying Visualization to the Search Workflow
3.1.3. Examples in Practice
- DG-Viz [35] is a visual analytics tool that uses multiple coordinated visualizations to display large collections of patient records. This allows users to observe data distributions across many documents at once. By leveraging visualization, DG-Viz can show far more records simultaneously than a list-based interface. Instead of reading each document, users can quickly select a region or cluster in a visualization that meets certain criteria (Figure 2A), greatly enhancing the parsing and exploratory querying process.
- OVERT-MED [6] is a tool designed specifically for search and triage in MEDLINE. It represents search results using a heatmap where each document is a horizontal bar and color saturation indicates the presence of query terms (Figure 3). This visual encoding allows users to perform a rapid perceptual scan to identify which results are most relevant to their query, facilitating rapid parsing and comparison without reading any text.
- EEEvis [17] is another tool designed for MEDLINE search that, in conjunction with a standard list view, provides multiple visualizations. Its co-authorship network (Figure 4) is particularly useful for analysis, as it shows the prevalence and connections between authors in the search results. This visualization allows users to quickly identify key researchers and collaborative groups, a task that would otherwise require significant time and manual effort.
3.2. Interaction
3.2.1. The Problem: The Interaction Deficit in MEDLINE Search
3.2.2. The Solution: Building a Richer Interactive Environment
- Provide proactive feedback. Interfaces can be designed to signal the expected outcome of an action before it is taken, a concept known as sensitivity encoding [47]. In a search context, the tool could show a real-time estimate of the result count as a user types a query. This immediate feedback helps users gauge a query’s restrictiveness and adjust it on the fly, reducing trial-and-error.
- Empower user control and manipulation. Search should not be a one-way street. Richer interactions allow users to actively organize, annotate, and manipulate the result space. The ability to visually group similar documents, mark items as “seen” or “relevant”, hide irrelevant results, and rearrange items based on personal criteria allows users to externalize their mental model and manage the information in a way that suits their specific task [44].
- Support complex querying and analysis. Beyond a simple text box, advanced interfaces can allow users to build queries visually or facet-by-facet. For analysis, interactions like selecting a document to see its relationship to all others, or highlighting a term to see its distribution across the entire result set, enable a much deeper engagement with the information than simply clicking a link.
3.2.3. Examples in Practice
- OVERT-MED [6] exemplifies the principle of proactive feedback. As a user builds a query, the system displays a real-time estimate of the result count. This simple but powerful interaction gives the user immediate feedback on their query building, helping them avoid queries that are either too broad or too narrow before they even run the search.
- NameClarifier [14], a tool for disambiguating homonymous and synonymous author names in document sets, demonstrates the power of user control. It provides interactive features that let users group search results by inferred author identity, eliminate irrelevant groups, and iteratively refine how the system resolves ambiguity. By supporting these additional interactions, NameClarifier enhances the user’s ability to parse and make sense of complex, ambiguous results.
- ChatRetriever [48] uses LLMs to allow users to engage in conversational search. Unlike traditional search, ChatRetriever allows users to articulate their information seeking needs through natural dialogue rather than formal query syntax. The system preserves dialogue context, enabling users to refine and extend their search as the discussion progresses. By leveraging this accumulated context, users’ prior interactions inform the system, empowering their ability to communicate and clarify their needs.
3.3. Machine Learning
3.3.1. The Problem: The Black Box, Usability Gap, and Hallucinations
3.3.2. The Solution: User-Centered Machine Learning Integration
- From keywords to intent. Instead of relying solely on keyword matching, ML can enable semantic search, which seeks to understand the concept behind a query [51]. If a user searches for “rough skin wound”, a semantic system can infer the user means “abrasion” and retrieve relevant documents even if they do not contain the exact keywords. This approach dramatically improves query building, especially for non-expert users who may not know the precise terminology [5].
- From opaque automation to human-in-the-loop. Rather than having ML models operate as a hidden black box, they can be designed to work collaboratively with the user. Techniques like active learning use human feedback to iteratively refine the model’s behavior. This keeps the user in control, builds trust, and leverages both human domain expertise and machine processing power to facilitate the parsing and analysis of large result sets.
3.3.3. Examples in Practice
- LitSuggest [69] exemplifies the shift from keywords to intent. Instead of keywords, users provide sets of “positive” (relevant) and “negative” (irrelevant) documents. An ML model learns from these examples to recommend new documents. This allows users to perform abstract queries like “find more like this”, fundamentally changing the query building process from a lexical task to a conceptual one.
- ASReview [15] is a powerful example of a “human-in-the-loop” system for systematic reviews. The tool uses active learning to assist with the parsing of thousands of articles. The user labels one document at a time as relevant or not, and the ML model instantly uses that feedback to re-rank the remaining documents, pushing the most likely relevant ones to the top. The user remains in complete control and can clearly see how their judgments guide the machine’s behavior, avoiding the black-box problem entirely.
- ALMANAC [63] is a retrieval-augmented generation framework for clinical information retrieval. To ground LLMs in factual context, ALMANAC augments user requests with a curated corpus of medical documents. Rather than relying on an LLM’s potentially unreliable training data, the framework leverages a data repository of trusted sources. Furthermore, ALMANAC requires the LLM to annotate its response with citations to the provided sources, allowing users to verify their authenticity and evaluate the quality of the response. This approach improves the factual reliability of responses compared with ungrounded LLMs. Users of tools that implement frameworks like ALMANAC retain the language processing power of LLMs while reducing their exposure to hallucinations.
3.4. Ontology
3.4.1. The Problem: The Vocabulary Mismatch
3.4.2. The Solution: The Ontology as a Semantic Bridge
- To enhance user queries. The most common application is for query expansion. An ontology can automatically augment a user’s query with synonyms and related concepts [75]. A search for “lungs” could be expanded to include “pulmonary” and “respiration”, retrieving a more comprehensive set of documents and relieving the user of the burden of brainstorming every possible term. This directly improves the query building sub-task.
- To enhance system understanding. Ontologies can also be used on the back end to make the system itself “smarter”. By indexing documents based on a structured ontology rather than just keywords, the system gains a much richer, more accurate understanding of its own content. This, in turn, can improve the performance of machine learning models used for ranking, recommendation, or entity recognition [18].
3.4.3. Examples in Practice
3.5. Triaging
3.5.1. The Problem: The Burden of Manual Triage
3.5.2. The Solution: Designing for the Triage Cycle
- High-level triage: This occurs during the initial parsing of results. The goal is to obtain a broad overview and quickly filter or group large sets of documents. This requires features that allow users to see the forest, not just the trees.
- Low-level triage: This occurs closer to the analysis phase. Here, the user examines the details of a smaller set of promising candidates to make final relevance judgments. This requires features that allow for focused comparison and inspection.
3.5.3. Examples in Practice
- VisualQUEST [11] was designed specifically to facilitate the triage cycle. Its interface is split into two linked views: a high-level view for grouping documents by topic or similarity, and a low-level view for showing detailed snippets of selected groups (Figure 5). This design directly maps to the two levels of triage, allowing users to efficiently switch between broad filtering and detailed examination.
- DocFlow [16], a system for systematic reviews, supports triage through a customizable “pipeline”. Users can build a multi-step process to filter documents (e.g., first by keyword, then by topic), which supports high-level triage. The system also provides detailed visualizations like scatter plots to explore smaller subsets, supporting low-level triage. By giving users interactive control over the entire filtering process, DocFlow empowers a more systematic and transparent triage workflow.
3.6. Progressive Disclosure
3.6.1. The Problem: Interface and Information Overload
3.6.2. The Solution: Aligning the Interface with the Search Stage
3.6.3. Examples in Practice
- VisualQUEST [11], a tool for literature search, is designed around this principle. Its interface components are tied to different stages of the search process (querying, high-level triage, low-level detail). As the user drills down from a broad overview to specific documents, the corresponding interface panels expand to show more detail while others recede into the background. This ensures that the user’s attention is always focused on the relevant tools for their current task.
- JARVIS [95] demonstrates how progressive disclosure can be used to manage recommendations. Instead of showing a long list of suggested queries upfront, JARVIS uses an ontology to gradually reveal related recommendations as the user searches and shows interest in certain topics. This avoids overwhelming the user while also providing contextual guidance at the moment it is most useful.
3.7. Evolutionary Design
3.7.1. The Problem: The Limits of Upfront Design
3.7.2. The Solution: Embracing an Iterative, User-Centered Process
3.7.3. Examples in Practice
3.8. Interconnections and Trade-Offs
4. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Abbreviations
LLM | Large language model |
ML | Machine learning |
MeSH | Medical Subject Headings |
References
- MEDLINE. Available online: https://www.nlm.nih.gov/medline/medline_home.html (accessed on 6 June 2025).
- PubMed. Available online: https://pubmed.ncbi.nlm.nih.gov/ (accessed on 6 June 2025).
- Wall, E.; Blaha, L.M.; Franklin, L.; Endert, A. Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Bias in Interactive Visual Analytics. In Proceedings of the 2017 IEEE Conference on Visual Analytics Science and Technology (VAST), Phoenix, AZ, USA, 3–6 October 2017; IEEE: New York, NY, USA, 2017; pp. 104–115. [Google Scholar]
- Herceg, P.M.; Allison, T.B.; Belvin, R.S.; Tzoukermann, E. Collaborative Exploratory Search for Information Filtering and Large-Scale Information Triage. J. Assoc. Inf. Sci. Technol. 2018, 69, 395–409. [Google Scholar] [CrossRef]
- Jin, Q.; Leaman, R.; Lu, Z. PubMed and Beyond: Biomedical Literature Search in the Age of Artificial Intelligence. eBioMedicine 2024, 100, 104988. [Google Scholar] [CrossRef] [PubMed]
- Demelo, J.; Parsons, P.; Sedig, K. Ontology-Driven Search and Triage: Design of a Web-Based Visual Interface for MEDLINE. JMIR Med. Inform. 2017, 5, e6918. [Google Scholar] [CrossRef] [PubMed]
- Salvador-Oliván, J.A.; Marco-Cuenca, G.; Arquero-Avilés, R. Development of an Efficient Search Filter to Retrieve Systematic Reviews from PubMed. J. Med. Libr. Assoc. 2021, 109, 561. [Google Scholar] [CrossRef] [PubMed]
- Islamaj Dogan, R.; Murray, G.C.; Névéol, A.; Lu, Z. Understanding PubMed® User Search Behavior Through Log Analysis. Database 2009, 2009, bap018. [Google Scholar] [CrossRef]
- Morshed, T.; Hayden, S. Google Versus PubMed: Comparison of Google and PubMed’s Search Tools for Answering Clinical Questions in the Emergency Department. Ann. Emerg. Med. 2020, 75, 408–415. [Google Scholar] [CrossRef]
- Gusenbauer, M.; Haddaway, N.R. Which Academic Search Systems Are Suitable for Systematic Reviews or Meta-Analyses? Evaluating Retrieval Qualities of Google Scholar, PubMed, and 26 Other Resources. Res. Synth. Methods 2020, 11, 181–217. [Google Scholar] [CrossRef]
- Demelo, J.; Sedig, K. Interfaces for Searching and Triaging Large Document Sets: An Ontology-Supported Visual Analytics Approach. Information 2021, 13, 8. [Google Scholar] [CrossRef]
- Demelo, J.; Sedig, K. Design of Generalized Search Interfaces for Health Informatics. Information 2021, 12, 317. [Google Scholar] [CrossRef]
- Cui, W. Visual Analytics: A Comprehensive Overview. IEEE Access 2019, 7, 81555–81573. [Google Scholar] [CrossRef]
- Shen, Q.; Wu, T.; Yang, H.; Wu, Y.; Qu, H.; Cui, W. Nameclarifier: A Visual Analytics System for Author Name Disambiguation. IEEE Trans. Vis. Comput. Graph. 2016, 23, 141–150. [Google Scholar] [CrossRef] [PubMed]
- Van De Schoot, R.; De Bruin, J.; Schram, R.; Zahedi, P.; De Boer, J.; Weijdema, F.; Kramer, B.; Huijts, M.; Hoogerwerf, M.; Ferdinands, G.; et al. An Open Source Machine Learning Framework for Efficient and Transparent Systematic Reviews. Nat. Mach. Intell. 2021, 3, 125–133. [Google Scholar] [CrossRef]
- Qiu, R.; Tu, Y.; Wang, Y.-S.; Yen, P.-Y.; Shen, H.-W. DocFlow: A Visual Analytics System for Question-Based Document Retrieval and Categorization. IEEE Trans. Vis. Comput. Graph. 2022, 30, 1533–1548. [Google Scholar] [CrossRef] [PubMed]
- Lee, J.-C.; Lee, B.J.; Park, C.; Song, H.; Ock, C.-Y.; Sung, H.; Woo, S.; Youn, Y.; Jung, K.; Jung, J.H.; et al. Efficacy Improvement in Searching MEDLINE Database Using a Novel PubMed Visual Analytic System: EEEvis. PLoS ONE 2023, 18, e0281422. [Google Scholar] [CrossRef] [PubMed]
- Xu, J.; Kim, S.; Song, M.; Jeong, M.; Kim, D.; Kang, J.; Rousseau, J.F.; Li, X.; Xu, W.; Torvik, V.I.; et al. Building a PubMed Knowledge Graph. Sci. Data 2020, 7, 205. [Google Scholar] [CrossRef]
- Zhu, Y.; Yuan, H.; Wang, S.; Liu, J.; Liu, W.; Deng, C.; Chen, H.; Liu, Z.; Dou, Z.; Wen, J.-R. Large Language Models for Information Retrieval: A Survey. arXiv 2023, arXiv:2308.07107. [Google Scholar] [CrossRef]
- Russell-Rose, T.; Shokraneh, F. Designing the Structured Search Experience: Rethinking the Query-Builder Paradigm. Weav. J. Libr. User Exp. 2020, 3. [Google Scholar] [CrossRef]
- Nitsche, M.; Nürnberger, A. QUEST: Querying Complex Information by Direct Manipulation. In Human Interface and the Management of Information: Information and Interaction Design, 15th International Conference, HCI International 2013, Las Vegas, NV, USA, July 21–26, 2013, Proceedings, Part I; Springer: Berlin/Heidelberg, Germany, 2013; pp. 240–249. [Google Scholar]
- Nowell, L.T.; France, R.K.; Hix, D.; Heath, L.S.; Fox, E.A. Visualizing Search Results: Some Alternatives to Query-Document Similarity. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Zurich, Switzerland, 18–22 August 1996; pp. 67–75. [Google Scholar]
- Peltonen, J.; Belorustceva, K.; Ruotsalo, T. Topic-Relevance Map: Visualization for Improving Search Result Comprehension. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, Limassol, Cyprus, 13–16 March 2017; pp. 611–622. [Google Scholar]
- Nguyen, T.; Zhang, J. A Novel Visualization Model for Web Search Results. IEEE Trans. Vis. Comput. Graph. 2006, 12, 981–988. [Google Scholar] [CrossRef]
- Heimerl, F.; Lohmann, S.; Lange, S.; Ertl, T. Word Cloud Explorer: Text Analytics Based on Word Clouds. In Proceedings of the 2014 47th Hawaii International Conference on System Sciences, Waikoloa, HI, USA, 6–9 January 2014; IEEE: New York, NY, USA, 2014; pp. 1833–1842. [Google Scholar]
- Mendoza, M.; Bonilla, S.; Noguera, C.; Cobos, C.; León, E. Extractive Single-Document Summarization Based on Genetic Operators and Guided Local Search. Expert Syst. Appl. 2014, 41, 4158–4169. [Google Scholar] [CrossRef]
- Hearst, M.A. Tilebars: Visualization of Term Distribution Information in Full Text Information Access. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, USA, 7–11 May 1995; pp. 59–66. [Google Scholar]
- Uren, V.; Lei, Y.; Lopez, V.; Liu, H.; Motta, E.; Giordanino, M. The Usability of Semantic Search Tools: A Review. Knowl. Eng. Rev. 2007, 22, 361–377. [Google Scholar] [CrossRef]
- McCallum, A.; Nigam, K.; Rennie, J.; Seymore, K. A Machine Learning Approach to Building Domain-Specific Search Engines. In Proceedings of the 16th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 31 July–6 August 1999; Volume 99, pp. 662–667. [Google Scholar]
- Larkin, J.H.; Simon, H.A. Why a Diagram Is (Sometimes) Worth Ten Thousand Words. Cogn. Sci. 1987, 11, 65–100. [Google Scholar] [CrossRef]
- Scaife, M.; Rogers, Y. External Cognition: How Do Graphical Representations Work? Int. J. Hum.-Comput. Stud. 1996, 45, 185–213. [Google Scholar] [CrossRef]
- Sedig, K.; Parsons, P. Interaction Design for Complex Cognitive Activities with Visual Representations: A Pattern-Based Approach. AIS Trans. Hum.-Comput. Interact. 2013, 5, 84–133. [Google Scholar] [CrossRef]
- Zhuang, S.; Zhuang, H.; Koopman, B.; Zuccon, G. A Setwise Approach for Effective and Highly Efficient Zero-Shot Ranking with Large Language Models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, DC, USA, 14–18 July 2024; pp. 38–47. [Google Scholar]
- Wu, Y.; Wan, Y.; Zhang, H.; Sui, Y.; Wei, W.; Zhao, W.; Xu, G.; Jin, H. Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study. Proc. ACM Manag. Data 2024, 2, 1–28. [Google Scholar] [CrossRef]
- Li, R.; Yin, C.; Yang, S.; Qian, B.; Zhang, P. Marrying Medical Domain Knowledge with Deep Learning on Electronic Health Records: A Deep Visual Analytics Approach. J. Med. Internet Res. 2020, 22, e20645. [Google Scholar] [CrossRef]
- Scells, H.; Zuccon, G. Searchrefiner: A Query Visualisation and Understanding Tool for Systematic Reviews. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Torino, Italy, 22–26 October 2018; pp. 1939–1942. [Google Scholar]
- Clarkson, E.; Desai, K.; Foley, J. Resultmaps: Visualization for Search Interfaces. IEEE Trans. Vis. Comput. Graph. 2009, 15, 1057–1064. [Google Scholar] [CrossRef]
- Görg, C.; Liu, Z.; Stasko, J. Reflections on the Evolution of the Jigsaw Visual Analytics System. Inf. Vis. 2014, 13, 336–345. [Google Scholar] [CrossRef]
- Liu, Y.-H.; Thomas, P.; Gedeon, T.; Rusnachenko, N. Search Interfaces for Biomedical Searching: How Do Gaze, User Perception, Search Behaviour and Search Performance Relate? In Proceedings of the 2022 Conference on Human Information Interaction and Retrieval, Regensburg, Germany, 14–18 March 2022; pp. 78–89. [Google Scholar]
- Aula, A.; Khan, R.M.; Guan, Z. How Does Search Behavior Change as Search Becomes More Difficult? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, GA, USA, 10–15 April 2010; pp. 35–44. [Google Scholar]
- Stolper, C.D.; Perer, A.; Gotz, D. Progressive Visual Analytics: User-Driven Visual Exploration of in-Progress Analytics. IEEE Trans. Vis. Comput. Graph. 2014, 20, 1653–1662. [Google Scholar] [CrossRef]
- Shao, L.; Silva, N.; Eggeling, E.; Schreck, T. Visual Exploration of Large Scatter Plot Matrices by Pattern Recommendation Based on Eye Tracking. In Proceedings of the 2017 ACM Workshop on Exploratory Search and Interactive Data Analytics, Limassol, Cyprus, 13 March 2017; pp. 9–16. [Google Scholar]
- Ola, O.; Sedig, K. The Challenge of Big Data in Public Health: An Opportunity for Visual Analytics. Online J. Public Health Inform. 2014, 5, 223. [Google Scholar] [PubMed]
- Fast, K.V.; Sedig, K. Interaction and the Epistemic Potential of Digital Libraries. Int. J. Digit. Libr. 2010, 11, 169–207. [Google Scholar] [CrossRef]
- Tenner, E. The Design of Everyday Things by Donald Norman. Technol. Cult. 2015, 56, 785–787. [Google Scholar] [CrossRef]
- Sedig, K.; Parsons, P.; Dittmer, M.; Ola, O. Beyond Information Access: Support for Complex Cognitive Activities in Public Health Informatics Tools. Online J. Public Health Inform. 2012, 4, 1–23. [Google Scholar] [CrossRef] [PubMed]
- Spence, R. Sensitivity Encoding to Support Information Space Navigation: A Design Guideline. Inf. Vis. 2002, 1, 120–129. [Google Scholar] [CrossRef]
- Mao, K.; Deng, C.; Chen, H.; Mo, F.; Liu, Z.; Sakai, T.; Dou, Z. ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA, 12–16 November 2024; pp. 1227–1240. [Google Scholar]
- Endert, A.; Ribarsky, W.; Turkay, C.; Wong, B.W.; Nabney, I.; Blanco, I.D.; Rossi, F. The State of the Art in Integrating Machine Learning into Visual Analytics. In Computer Graphics Forum; Wiley Online Library: Hoboken, NJ, USA, 2017; Volume 36, pp. 458–486. [Google Scholar]
- Portugal, I.; Alencar, P.; Cowan, D. The Use of Machine Learning Algorithms in Recommender Systems: A Systematic Review. Expert Syst. Appl. 2018, 97, 205–227. [Google Scholar] [CrossRef]
- Fang, F.; Zhang, B.-W.; Yin, X.-C. Semantic Sequential Query Expansion for Biomedical Article Search. IEEE Access 2018, 6, 45448–45457. [Google Scholar] [CrossRef]
- Aphinyanaphongs, Y.; Aliferis, C.F. Prospective Validation of Text Categorization Filters for Identifying High-Quality, Content-Specific Articles in MEDLINE. In AMIA Annual Symposium Proceedings; American Medical Informatics Association: Washington, DC, USA, 2006; Volume 2006, p. 6. [Google Scholar]
- Fiorini, N.; Canese, K.; Starchenko, G.; Kireev, E.; Kim, W.; Miller, V.; Osipov, M.; Kholodov, M.; Ismagilov, R.; Mohan, S.; et al. Best Match: New Relevance Search for PubMed. PLoS Biol. 2018, 16, e2005343. [Google Scholar] [CrossRef]
- Ma, C.; Zhang, W.E.; Guo, M.; Wang, H.; Sheng, Q.Z. Multi-Document Summarization via Deep Learning Techniques: A Survey. ACM Comput. Surv. 2022, 55, 1–37. [Google Scholar] [CrossRef]
- Khalid, S.; Khalil, T.; Nasreen, S. A Survey of Feature Selection and Feature Extraction Techniques in Machine Learning. In Proceedings of the 2014 Science and Information Conference, London, UK, 27–29 August 2014; IEEE: New York, NY, USA, 2014; pp. 372–378. [Google Scholar]
- Naveed, H.; Khan, A.U.; Qiu, S.; Saqib, M.; Anwar, S.; Usman, M.; Akhtar, N.; Barnes, N.; Mian, A. A Comprehensive Overview of Large Language Models. ACM Trans. Intell. Syst. Technol. 2025, 16, 1–72. [Google Scholar] [CrossRef]
- Tang, L.; Sun, Z.; Idnay, B.; Nestor, J.G.; Soroush, A.; Elias, P.A.; Xu, Z.; Ding, Y.; Durrett, G.; Rousseau, J.F.; et al. Evaluating Large Language Models on Medical Evidence Summarization. NPJ Digit. Med. 2023, 6, 158. [Google Scholar] [CrossRef] [PubMed]
- Van Veen, D.; Van Uden, C.; Blankemeier, L.; Delbrouck, J.-B.; Aali, A.; Bluethgen, C.; Pareek, A.; Polacin, M.; Reis, E.P.; Seehofnerová, A.; et al. Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization. Nat. Med. 2024, 30, 1134–1142. [Google Scholar] [CrossRef]
- Ntinopoulos, V.; Biefer, H.R.C.; Tudorache, I.; Papadopoulos, N.; Odavic, D.; Risteski, P.; Haeussler, A.; Dzemali, O. Large Language Models for Data Extraction from Unstructured and Semi-Structured Electronic Health Records: A Multiple Model Performance Evaluation. BMJ Health Care Inform. 2025, 32, e101139. [Google Scholar] [CrossRef]
- Jagerman, R.; Zhuang, H.; Qin, Z.; Wang, X.; Bendersky, M. Query Expansion by Prompting Large Language Models. arXiv 2023, arXiv:2305.03653. [Google Scholar] [CrossRef]
- Agrawal, G.; Kumarage, T.; Alghamdi, Z.; Liu, H. Can Knowledge Graphs Reduce Hallucinations in LLMs?: A Survey. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers); Association for Computational Linguistics: Mexico City, Mexico, 2024; pp. 3947–3960. [Google Scholar]
- Tan, X.; Wang, X.; Liu, Q.; Xu, X.; Yuan, X.; Zhang, W. Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning. In Proceedings of the ACM on Web Conference 2025, Sydney, Australia, 28 April–2 May 2025; pp. 3505–3522. [Google Scholar]
- Zakka, C.; Shad, R.; Chaurasia, A.; Dalal, A.R.; Kim, J.L.; Moor, M.; Fong, R.; Phillips, C.; Alexander, K.; Ashley, E.; et al. Almanac—Retrieval-Augmented Language Models for Clinical Medicine. Nejm Ai 2024, 1, AIoa2300068. [Google Scholar] [CrossRef]
- Kiester, L.; Turp, C. Artificial Intelligence Behind the Scenes: PubMed’s Best Match Algorithm. J. Med. Libr. Assoc. 2022, 110, 15. [Google Scholar] [CrossRef]
- Cierco Jimenez, R.; Lee, T.; Rosillo, N.; Cordova, R.; Cree, I.A.; Gonzalez, A.; Indave Ruiz, B.I. Machine Learning Computational Tools to Assist the Performance of Systematic Reviews: A Mapping Review. BMC Med. Res. Methodol. 2022, 22, 322. [Google Scholar] [CrossRef]
- Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y.J.; Madotto, A.; Fung, P. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 2023, 55, 1–38. [Google Scholar] [CrossRef]
- Hu, J.-M.; Liu, F.-C.; Chu, C.-M.; Chang, Y.-T. Health Care Trainees’ and Professionals’ Perceptions of ChatGPT in Improving Medical Knowledge Training: Rapid Survey Study. J. Med. Internet Res. 2023, 25, e49385. [Google Scholar] [CrossRef] [PubMed]
- Spotnitz, M.; Idnay, B.; Gordon, E.R.; Shyu, R.; Zhang, G.; Liu, C.; Cimino, J.J.; Weng, C. A Survey of Clinicians’ Views of the Utility of Large Language Models. Appl. Clin. Inform. 2024, 15, 306–312. [Google Scholar] [CrossRef] [PubMed]
- Allot, A.; Lee, K.; Chen, Q.; Luo, L.; Lu, Z. LitSuggest: A Web-Based System for Literature Recommendation and Curation Using Machine Learning. Nucleic Acids Res. 2021, 49, W352–W358. [Google Scholar] [CrossRef]
- Arp, R.; Smith, B.; Spear, A.D. Building Ontologies with Basic Formal Ontology; The MIT Press: Cambridge, MA, USA, 2015. [Google Scholar]
- Gargano, M.A.; Matentzoglu, N.; Coleman, B.; Addo-Lartey, E.B.; Anagnostopoulos, A.V.; Anderton, J.; Avillach, P.; Bagley, A.M.; Bakštein, E.; Balhoff, J.P.; et al. The Human Phenotype Ontology in 2024: Phenotypes Around the World. Nucleic Acids Res. 2024, 52, D1333–D1346. [Google Scholar] [CrossRef] [PubMed]
- Ashburner, M.; Ball, C.A.; Blake, J.A.; Botstein, D.; Butler, H.; Cherry, J.M.; Davis, A.P.; Dolinski, K.; Dwight, S.S.; Eppig, J.T.; et al. Gene Ontology: Tool for the Unification of Biology. Nat. Genet. 2000, 25, 25–29. [Google Scholar] [CrossRef] [PubMed]
- Doms, A.; Schroeder, M. GoPubMed: Exploring PubMed with the Gene Ontology. Nucleic Acids Res. 2005, 33, W783–W786. [Google Scholar] [CrossRef]
- Trieschnigg, D.; Pezik, P.; Lee, V.; De Jong, F.; Kraaij, W.; Rebholz-Schuhmann, D. MeSH up: Effective MeSH Text Classification for Improved Document Retrieval. Bioinformatics 2009, 25, 1412–1418. [Google Scholar] [CrossRef] [PubMed]
- Bhogal, J.; MacFarlane, A.; Smith, P. A Review of Ontology Based Query Expansion. Inf. Process. Manag. 2007, 43, 866–886. [Google Scholar] [CrossRef]
- Gracia, J.; Trillo, R.; Espinoza, M.; Mena, E. Querying the Web: A Multiontology Disambiguation Method. In Proceedings of the 6th International Conference on Web Engineering, Palo Alto, CA, USA, 11–14 July 2006; pp. 241–248. [Google Scholar]
- Asim, M.N.; Wasim, M.; Khan, M.U.G.; Mahmood, N.; Mahmood, W. The Use of Ontology in Retrieval: A Study on Textual, Multilingual, and Multimedia Retrieval. IEEE Access 2019, 7, 21662–21686. [Google Scholar] [CrossRef]
- de Silva, N.; Dou, D.; Huang, J. Discovering Inconsistencies in Pubmed Abstracts Through Ontology-Based Information Extraction. In Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Boston, MA, USA, 20–23 August 2017; pp. 362–371. [Google Scholar]
- Furnas, G.W.; Landauer, T.K.; Gomez, L.M.; Dumais, S.T. The Vocabulary Problem in Human-System Communication. Commun. ACM 1987, 30, 964–971. [Google Scholar] [CrossRef]
- Salvador-Oliván, J.A.; Marco-Cuenca, G.; Arquero-Avilés, R. Errors in Search Strategies Used in Systematic Reviews and Their Effects on Information Retrieval. J. Med. Libr. Assoc. 2019, 107, 210. [Google Scholar] [CrossRef]
- Wang, J.Z.; Zhang, Y.; Dong, L.; Li, L.; Srimani, P.K.; Yu, P.S. G-Bean: An Ontology-Graph Based Web Tool for Biomedical Literature Retrieval. BMC Bioinform. 2014, 15, S1. [Google Scholar] [CrossRef]
- Loizides, F.; Buchanan, G. An Empirical Study of User Navigation During Document Triage. In Research and Advanced Technology for Digital Libraries: 13th European Conference. ECDL 2009, Corfu, Greece, September 27–October 2, 2009, Proceedings; Springer: Berlin/Heidelberg, Germany, 2009; pp. 138–149. [Google Scholar]
- Loizides, F.; Buchanan, G. Towards a Framework for Human (Manual) Information Retrieval. In Multidisciplinary Information Retrieval: 6th Information Retrieval Facility Conference, IRFC 2013, Limassol, Cyprus, October 7–9, 2013, Proceedings; Springer: Berlin/Heidelberg, Germany, 2013; pp. 87–98. [Google Scholar]
- Jonker, D.; Wright, W.; Schroh, D.; Proulx, P.; Cort, B. Information Triage with TRIST. In Proceedings of the 2005 Intelligence Analysis Conference, Washington, DC, USA, 2–6 May 2005; pp. 2–4. [Google Scholar]
- Macskassy, S.A.; Provost, F. Intelligent Information Triage. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA, USA, 9–13 September 2001; pp. 318–326. [Google Scholar]
- Buchanan, G.; Owen, T. Improving Skim Reading for Document Triage. In Proceedings of the Second International Symposium on Information Interaction in Context, London, UK, 14–17 October 2008; pp. 83–88. [Google Scholar]
- Ouzzani, M.; Hammady, H.; Fedorowicz, Z.; Elmagarmid, A. Rayyan—A Web and Mobile App for Systematic Reviews. Syst. Rev. 2016, 5, 210. [Google Scholar] [CrossRef]
- Badi, R.; Bae, S.; Moore, J.M.; Meintanis, K.; Zacchi, A.; Hsieh, H.; Shipman, F.; Marshall, C.C. Recognizing User Interest and Document Value from Reading and Organizing Activities in Document Triage. In Proceedings of the 11th International Conference on Intelligent User Interfaces, Sydney, Australia, 29 January–1 February 2006; pp. 218–225. [Google Scholar]
- Rathbone, J.; Hoffmann, T.; Glasziou, P. Faster Title and Abstract Screening? Evaluating Abstrackr, a Semi-Automated Online Screening Program for Systematic Reviewers. Syst. Rev. 2015, 4, 80. [Google Scholar] [CrossRef]
- Springer, A.; Whittaker, S. Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? ACM Trans. Interact. Intell. Syst. (TiiS) 2020, 10, 1–32. [Google Scholar] [CrossRef]
- Chuang, J.; Ramage, D.; Manning, C.; Heer, J. Interpretation and Trust: Designing Model-Driven Visualizations for Text Analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, USA, 5–10 May 2012; pp. 443–452. [Google Scholar]
- Phan, D.; Paepcke, A.; Winograd, T. Progressive Multiples for Communication-Minded Visualization. In Proceedings of the Proceedings of Graphics Interface 2007, Montreal, QC, Canada, 28–30 May 2007; pp. 225–232. [Google Scholar]
- Springer, A.; Whittaker, S. Progressive Disclosure: Empirically Motivated Approaches to Designing Effective Transparency. In Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA, 17–20 March 2019; pp. 107–120. [Google Scholar]
- Oulasvirta, A.; Hukkinen, J.P.; Schwartz, B. When More Is Less: The Paradox of Choice in Search Engine Use. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Boston, MA, USA, 19–23 July 2009; pp. 516–523. [Google Scholar]
- Ribeiro, D.S.; de Sousa, A.G.; de Almeida, R.B.; Thompson Furtado, P.H.; Côrtes Vieira Lopes, H.; Barbosa, S.D.J. Exploring Ontology-Based Information Through the Progressive Disclosure of Visual Answers to Related Queries. In Human Interface and the Management of Information. Designing Information: Thematic Area, HIMI 2020, Held as Part of the 22nd International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part I; Springer: Berlin/Heidelberg, Germany, 2020; pp. 104–124. [Google Scholar]
- Stouffs, R.; Rafiq, Y. Generative and Evolutionary Design Exploration. AI EDAM 2015, 29, 329–331. [Google Scholar] [CrossRef]
- Guerrero-García, J. Evolutionary Design of User Interfaces for Workflow Information Systems. Sci. Comput. Program. 2014, 86, 89–102. [Google Scholar] [CrossRef]
- Schleimer, E.; Pearce, J.; Barnecut, A.; Rowles, W.; Lizee, A.; Klein, A.; Block, V.J.; Santaniello, A.; Renschen, A.; Gomez, R.; et al. A Precision Medicine Tool for Patients with Multiple Sclerosis (the Open MS BioScreen): Human-Centered Design and Development. J. Med. Internet Res. 2020, 22, e15605. [Google Scholar] [CrossRef] [PubMed]
- Fiorini, N.; Canese, K.; Bryzgunov, R.; Radetska, I.; Gindulyte, A.; Latterner, M.; Miller, V.; Osipov, M.; Kholodov, M.; Starchenko, G.; et al. PubMed Labs: An Experimental System for Improving Biomedical Literature Search. Database 2018, 2018, bay094. [Google Scholar] [CrossRef] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhurov, V.; Sedig, K.; Milani, M. Beyond the List: A Framework for the Design of Next-Generation MEDLINE Search Tools. Data 2025, 10, 167. https://doi.org/10.3390/data10100167
Zhurov V, Sedig K, Milani M. Beyond the List: A Framework for the Design of Next-Generation MEDLINE Search Tools. Data. 2025; 10(10):167. https://doi.org/10.3390/data10100167
Chicago/Turabian StyleZhurov, Vladimir, Kamran Sedig, and Mostafa Milani. 2025. "Beyond the List: A Framework for the Design of Next-Generation MEDLINE Search Tools" Data 10, no. 10: 167. https://doi.org/10.3390/data10100167
APA StyleZhurov, V., Sedig, K., & Milani, M. (2025). Beyond the List: A Framework for the Design of Next-Generation MEDLINE Search Tools. Data, 10(10), 167. https://doi.org/10.3390/data10100167