Enhancing Education Through Generative AI: A Multimodal Approach to Semantic Search and Authentic Learning

In contemporary education, learners face the challenge of navigating an overwhelming abundance of information. Traditional search methods, often limited to keyword matching, fail to capture the nuanced meaning and relationships within educational materials. Our multimodal approach combines Sentence Transformer for text and Inception V3 for images to generate vector embeddings for textbooks which are stored in an Elasticsearch database. Learners’ queries again are converted to vector embeddings which are matched through cosine similarity with stored embeddings, resulting in retrieval of relevant material which is ranked and then synthesized using large language model (LLM) APIs. The approach retrieves answers based on semantic search rather than keywords. The system also integrates GenAI capabilities separately, specifically leveraging LLM APIs, to generate context-aware answers to user-posed questions at varying levels of complexity, e.g., beginner, intermediate, and advanced. Through comprehensive evaluation, we demonstrate the system’s ability to retrieve coherent answers across multiple sources, offering significant advancements in cross-text and cross-modal retrieval tasks. This work also contributes to the international discourse on ethical GenAI integration in curricula and fosters a collaborative human–AI learning ecosystem.