The Rise of Foundation Models: Opportunities, Technology, Applications, Challenges, Recent Trends, and Future Directions

Abstract

Foundation models (FMs) have become a paradigm shift in the field of artificial intelligence, allowing one large-scale pretrained model to be customized for a broad set of downstream tasks using very little task-specific data. These models, which include GPT, CLIP, BERT, and vision transformers, have altered the scope of transfer learning and multimodal understanding and are built on top of enormous datasets and self-supervised learning. The paper provides a broad view of the modern state of foundation models, with an emphasis on their technological foundation, training, and cross-domain use in fields like natural language processing, computer vision, healthcare, robotics and scientific discovery. We also explore the main opportunities that FMs offer, as well as state-of-the-art methods and techniques for the development of foundation models. we discuss their applications in natural language processing, computer vision, healthcare, etc. Furthermore, their limitations and challenges are also investigated. Lastly, future prospects are discussed so that professionals and scientists obtain a better understanding of the importance of foundation models for addressing their research goals.

1. Introduction

1.1. Background and Motivation

Artificial intelligence is transforming industries by enhancing efficiency, improving decision making, and enabling innovation across various domains. Deep learning has significantly boosted the development of predictive models for multi-dimensional data like text and images, which was difficult for traditional machine learning algorithms [1,2]. Pretraining the model is a very common and central approach in deep learning, used for transfer learning, where a model is trained on a related task to learn useful features, and then this pretrained model is fine-tuned to perform well for the specific task of interest. Transfer learning has enabled foundation models and made them possible. For at least a decade, transfer learning with labeled datasets has been widely practiced; pretraining on the Imagenet dataset for image classification is a common example in the computer vision field [3]. Nevertheless, the cost of annotation or labeling puts a constraint on the advantages of pretraining. On the other hand, self-supervised learning automatically generates pretraining from unlabeled data [4,5]. Vision transformers (ViTs) and CNNs [6,7,8] have been used for the training of predictive models using supervised learning techniques to perform multiple tasks efficiently. Lately, self-supervised learning has transformed the field by allowing deep neural networks to be trained on very large unlabeled datasets, achieving high performance compared to previous supervised learning approaches. They are also more scalable and potentially more valuable than those trained on restricted labeled data. Models trained this way are known as foundation models, which can be used for carrying out multiple downstream tasks with little or no fine-tuning [9,10]. Foundation models have rapidly gained prominence, particularly in the field of natural language and computer vision. These models open new avenues in the field of generative artificial intelligence and machine learning. The term “foundation models” was first coined by a team of researchers at the Stanford Institute for Human-Centered AI [11]. Foundation models are large-scale artificial intelligence-based models that are trained on a plethora of unlabeled diverse data using mostly self-supervised learning or semi-supervised learning approaches. They demonstrate promising potential in learning the intrinsic feature representations of data. This training methodology caters to the diverse capabilities of the models, enabling them to excel in a wide range of tasks, for example, computer vision, object detection and classification, information retrieval from images, image segmentation and classification, natural language processing, text generation, image-to-text and text-to-image generation, graph classification, and many more [12]. Figure 1 shows the architecture of the foundation model. Foundation models can have various input modalities such as text, images, and structured or unstructured data, as shown in Figure 1. These modalities go through different preprocessing steps, and the foundation model is then trained on this huge amount of data using self-supervised learning approaches.

As stated earlier, foundation models are usually trained using self-supervision techniques on a large amount of data taken from online resources. These general-purpose deep models have more than 300 million parameters. They can learn and address a number of common problems, achieving high performance, although they are not trained or designed for dedicated tasks. These datasets hold large amounts of general information, making them capable and adaptable for handling various common tasks, yet they are unable to incorporate all the expert knowledge and technical details of certain domains. Therefore, they lack the required details to execute domain-specific tasks. However, they can be fine-tuned to perform some domain-specific or downstream tasks [13]. Big companies like Google, OpenAI, and Meta have made substantial funding and investments for the development of foundation models [14]. Developing a foundation model from scratch costs millions of dollars; however, it is worth it, as they are very useful and beneficial in the long run. Building custom ML applications has now become easier by using pre-trained foundation models, as developers no longer have to develop unique ML models from scratch. Instead, they can use a pre-trained foundation model, which is cost-effective and quick. Some of the well-known examples of foundation models are the Llama series, the GPT series, BERT, and CLIP. These models have been trained on billions of parameters, manifesting impressive results without reliance on domain-specific data or parameter optimization.

1.2. Characteristics of Foundation Models

Foundation models usually have the following characteristics:

1.2.1. Pre-Training Processes

• Extensive datasets: Using comprehensive datasets from a wide range of domains (i.e., books, websites, articles, images) to train the model, providing it with broad knowledge.
• Self-supervised learning: They mostly use self-supervised learning techniques to learn from large unlabeled datasets like predicting a missing word based on the preceding word in a sequence.

1.2.2. Scale and Architecture

• Parameters: A foundation model consists of a vast number of parameters, which allows it to incorporate insights derived from the massive data into its design [9].
• Adaptability: The model has the ability to adapt and adjust its input data and processing tasks based on the different and changing requirements [15].
• Ability to generalize: The model presents the ability to generalize, enabling it to perform well even when faced with new or unfamiliar data [15].
• Deep learning architecture: They use advanced deep learning frameworks, usually transformers, that are suitable for handling sequential data like text, etc. [16].

1.2.3. Fine-Tuning and Transferability

• Fine-tuning: Foundation models can be fine-tuned on smaller datasets by adjusting their weights to achieve optimal performance, after pretraining, like sentiment analysis or image classification. One kind of transfer learning is pretraining, which is used in the GPT-n series of LLMs [17].
• Task-specific adaptation: By fine-tuning the model, it can perform domain-specific tasks more effectively, making it more resource efficient in comparison to training from scratch [18]. Figure 2 represents the main characteristics of foundation models.

2. Methodology

The primary goal of this systematic literature review is to find out the current state-of-the-art approaches based on foundation models and their applications, challenges, and future. This study demonstrates a comprehensive review of state-of-the-art foundation models, aiming to provide the latest and up-to-date information on foundation models and their potential applications. We started our review work in a systematic way, and the steps are as follows.

2.1. Research Questions

For conducting any kind of systematic review, the formulation of research questions is very important. These questions provide guidelines for data collection, curation, and analysis. The research questions and systematic reviews (PICOC) framework helps formulate and establish these questions. Consequently, it provides guidance to conduct the research in a very conducive way, gaining the interest of researchers and other readers.

Table 1. PICOC framework for foundation model study.

Population	Intervention	Comparison	Outcomes	Context
Foundation models, general-purpose foundation models, large foundation models, multi-modal models, transformer-based models.	Trends in foundation models, training techniques, fine-tuning methods, pretraining datasets, model architecture, evaluation metrics.	Investigate various foundation models	Performance metrics, application success, scalability, ethical considerations.	Studies in AI research, industry applications, academia, and various datasets.

shows the PICOC framework used in this study.

Population, intervention, comparison, outcomes and context are the key factors in the PICOC framework. Population refers to the area of interest or target for the investigation. In this research study, the population refers to foundation models. Intervention means investigating and exploring the targeted area of research while specifying different aspects. We investigate foundation models, trends in foundation models, different foundation models’ architecture, training techniques, pretraining datasets, and foundation models in different fields like computational pathology, natural languages. Comparison means to compare the population with others. Outcomes means the output or the results of the intervention, and lastly, context refers to the setting or environment in which the investigation takes place. Therefore, considering the PICOC framework, the objective of this research study is to identify recent trends in foundation models and their challenges.

Table 2. Research questions and motivations for foundation model study.

Code	Research Questions	Motivations
RQ1	What are the opportunities, capabilities, and potentials of foundation models?	To explore and identify the potential, advantages, and capabilities of foundation models.
RQ2	What are the state-of-the-art technologies, techniques, and methods used for devel-oping foundation models?	To discover the best approaches for developing foundation models that demonstrate encouraging performance.
RQ3	What are the applications and use cases of foundation models?	To examine the areas where foundation models are being applied.
RQ4	What are the challenges, limitations, and obstacles in developing foundation models?	To explore the challenges and limitations associated with building foundation models.
RQ5	What are the recent trends and future perspectives of foundation models?	To investigate the current trends and emerging applications of foundation models.

represents the research questions and motivations used in this study.

2.2. Search Strategy

A proper and systematic search strategy must be used to help ensure the thoroughness of the research. Figure 3 shows an overview of our study. We used certain keywords and strings related to our research questions to collect articles from digital libraries through automatic search and manual inquiry. Some examples of the string formulation are as follows: (“Foundation models” OR “large models” OR “Large-scale models”) AND (“Opportunities” OR “Benefits” OR “Potentials” OR “Capabilities” OR “Advantages”) AND (“Technology” OR “Approaches” OR “Techniques” OR “Methods”) AND (“Applications” OR “Use Cases” OR “ Domains” OR “Implementations”) AND (“Challenges” OR “Limitations” OR “Hurdles” OR “Barriers” OR “Obstacles”) AND (“Recent Trends” OR “ Current Developments” OR “Current Trends” OR “Innovations”) AND (“Future Directions” OR “Future Research” OR “Future Prospects” OR “Emerging Trends”). We also collected bibliographic records using the OpenAlex API for scholarly works related to foundation models. We included keywords like “foundation models,” “large language models,” “multimodal models,” and several area-specific terms (e.g., “foundation models in healthcare,” “foundation models in computer vision”). For each keyword, we retrieved all indexed works published between 2020 and 2025. We used the OpenAlex cursor-based pagination API to ensure exhaustive retrieval of results.

2.3. Study Selection

We defined certain selection criteria and chose appropriate studies. The literature written in languages other than English was excluded from our consideration. We also ensured full access to the articles. Furthermore, the content should be in the domain of our research questions, for example foundation models, their new trends, capabilities, technologies, challenges, and emerging and future perspectives. Figure 4 shows the domain trends of the foundation model used in various fields from 2020 to 2025. Likewise, Figure 5 shows the cumulative papers on foundation models. There has been a significant increase in research on foundation models during the last five years. Figure 6 shows the number of papers on foundation models based on different domains. Figure 7 demonstrates the number of papers on foundation models by year. The analysis shows that in 2024, more than 2500 papers were published on foundation models. Figure 8 represents the knowledge graph on foundation models.

• Opportunities and capabilities of foundation models over traditional models

Foundation models possess a range of abilities and salient features, making them a powerful tool for downstream tasks. They are also capable of learning several new skills during their training phase, which helps them perform tasks in various domains like vision, interaction, robotics, reasoning, and linguistics. These models are trained using self-supervised learning techniques, which are quite similar to the theoretical concept of understanding.

Therefore, these models are able to understand things better as they continue to learn. The tasks foundation models can perform in various domains are as follows:

• Linguistics: Understanding language and generating text.
• Vision: Interpreting and processing visuals, i.e., images and videos.
• Robotics: Having the capability to impact the physical environment through robotic technology.
• Interaction: Communicating, collaborating, and interacting with humans.
• Reasoning: Capable of reasoning and searching, which allows them to find solutions and gather data.

2.4. Difference Between Foundation and Non-Foundation Models

AI models demonstrate different characteristics like expressiveness, which enables a model with increased complexity to represent a wider range of relationships, and scalability, the ability to perform better with more data [19]. The architecture of foundation models is based on vision transformers, which could be used to handle multiple tasks, attaining top-tier performance. They do not need to be trained on labeled data; however, they require a very large amount of data to perform; their performance can range from medium to high on untrained tasks. Thus, foundation models provide more expressiveness and scalability due to their larger size, more training data, and capability to use parallelized training techniques. A single generic learning algorithm can power a variety of applications due to homogenization; this same method could be used in many domains. Almost all of the advanced and state-of-the-art NLP (Natural Language Processing) models are derived from a small number of foundation models, as stated by [20,21]. The Stanford Institute HAI report [11] also supports this. On the contrary, non-foundation models, i.e., CNNs, do not perform well on tasks they have not been trained for but excel in specialized or adapted tasks. They are based on convolutional neural networks of medium to large size and are designed for single-task applications. Both foundation and non-foundation models depend on embeddings for understanding and processing input data. These are basically compressed representations or tensors or vectors that enable the model to identify patterns and relationships in the data. Their quality determines the overall performance accuracy of the model for downstream tasks.

Table 3. Comparison between foundation model and non-foundation models.

Features	Foundation Models	Non-Foundation Models
Reliance on labeled data	Low reliance during pretraining; down-stream fine-tuning typically requires labeled data [11]	High reliance on labeled data for both training and fine-tuning [22,23]
Training data	Massive, diverse, multi-domain datasets [15]	Medium to small, task-specific datasets
Model size	Very large (billions of parameters) [24]	Smaller (thousands to millions of parameters) [25]
Architecture	Dominantly transformer-based [26]	Varied (e.g., CNNs, RNNs, small transformers) [23]
Training method	Highly parallelizable large-scale pretraining [24]	Standard sequential or batch-based training [23]
Expressiveness	Highly expressive and capable of capturing broad patterns	More limited expressiveness, domain/task-specific
Scalability	Highly scalable due to architecture and training framework [15]	Less scalable; training costs rise sharply with model size [27]
Complexity	High computational and architectural complexity [26]	Lower overall complexity [25]
Embedding generation	Learns universal embeddings during pretraining [28]	Embedding generation depends on model type
Adaptability	Strong cross-task generalization and transfer learning [28]	Primarily specialized for a single downstream task [23]

shows the comparison between foundation models and non-foundation models. Furthermore,

Table 4. Comparative analysis of traditional NLP approaches and modern NLP (foundation models).

Characteristic	Traditional NLP Models	Foundation Models
Model Type	Task-specific models are used to perform each sub-task like token segmentation and syntactic parsing.	A single foundation model is fine-tuned to perform multiple tasks such as sentiment classification, translation, and summarization.
Architecture	Pipeline with separate task-specific models for each unique task, such as tokenization or coreference resolution.	One model fine-tuned with small datasets for each individual task.
Performance	Moderate performance: Traditional NLP question-answering models achieved accuracy of 73.1% for answering open-ended science questions [29].	Higher performance: A fine-tuned foundation model achieved 91.6% accuracy for the same exam in the following year [29].
Language Generation	Considered very difficult and achievable only through linguistic subtasks [30].	Achieved by models like GPT-3, which predict the next word in a sequence [31].
Speech Recognition	Separate models were needed for speech-related tasks such as ASR (Automatic Speech Recognition).	Foundation models like wav2vec 2.0 are trained on large speech audio datasets and then fine-tuned for ASR tasks [32].
Impact on NLP	Narrow scope, focusing on individual sub-tasks.	Broad scope, enabling multiple related tasks within a unified frame-work.
Leveraging Linguistic Diversity	Limited support for non-English languages, dialects, and linguistic variations.	Can handle multiple languages using models like XLM-R, mT5, and mBERT to enable cross-lingual flow from high- to low-resource languages [33,34,35].
Multilingual Model Adaptation	Task-specific models struggle with cross-lingual adaptation.	Foundation models like GShard handle large-scale multilingual tasks, significantly improving low-resource languages [36].
Real-world Language Understanding	Traditional models lack human-like contextual understanding.	RoBERTa incorporates real-world context into language learning, mimicking human-level under-standing [37].

and

Table 5. Comparative analysis of traditional supervised model vs. foundation model in Computer Vision.

Feature	Traditional Supervised Model	Foundation Models
Architecture	Primarily convolutional neural networks (CNNs).	Vision transformers (ViTs), multimodal architectures (CLIP, DALL-E).
Annotation Depen- dence	Requires large, annotated datasets, which limit scalability and diversity.	Reduced annotation requirement by leveraging large unlabeled data in the pretraining phase.
Scalability	Hard to adapt to new tasks without labeled data.	Can be adapted to large data and models.
Flexibility	Only limited to visual tasks.	Multimodal: text, images, and other modalities.
Generalization	Limited task generalization.	Generalization across various tasks, supports transfer learning.
Core Tasks	Image classification, object detection, and segmentation.	Traditional tasks plus image generation, multimodal reasoning, visual question answering, and others.
Performance	State-of-the-art performance for specific tasks.	Medium to state-of-the-art performance for various tasks with little fine-tuning, including unseen tasks.
Representation Learning	Needs fully supervised or hand-crafted feature learning.	Uses self-supervised or contrastive learning to acquire high-level feature representation.
Efficiency	Demands a large amount of labeled data and processing power.	Self-supervision and transfer learning make data utilization more efficient but computationally expensive.
Application	Suitable for specific applications like security and autonomous driving.	Image synthesis (DALL-E), common-sense reasoning, and medical imaging.

represent the comparative analysis of foundation models with traditional NLP and supervised learning approaches, respectively.

3. State-of-the-Art Methods and Techniques for the Development of the Foundation Model

Many specialized and sophisticated applications are built upon foundation models. The development of foundation models has rapidly improved through the application of large-scale pretraining and fine-tuning methodologies across a wide range of domains. They generally specialize in task generalization capability and have re-created many domains related to natural language processing, computer vision, and robotics.

Table 6. Summary of foundation model applications across various domains.

Objective	Dataset	Data Type	Contribution	Model Used	Reference
Auto Speech Recognition	Libri-Light, LibriSpeech, TIMIT	Voice data	Improved speech representations	Encoder–Decoder	[38]
Large-Scale Language Understanding	Mixture of 780B Text, English NLP, BIG-bench Reasoning, Code, etc.	Mixed data	Scaling Few-Shot Performance, multilingual and code understanding	Unsupervised	[39]
Chatbots (ChatGPT 3.5, 4) evaluation against Family Medicine residents	MCQs	Multiple-choice questions (MCQs)	GPT-4 performed the best	ChatGPT (3.5 and 4)	[40]
Vision-Language Tasks	VQA-RAD, MIMIC-CXR, Open-I	Multimodal (image–text)	Multi-modal understanding of text–image pairs	ResNet-50, BERT	[41]
Classification, Segmentation, Image-Text Retrieval	SIIM Pneu- mothorax, CheXpert, RSNA Pneu-monia	Image–text	Improved performance	ResNet, Bio-ClinicalBERT	[42]
Text Generation	Common Crawl, C4, Github, Wikipedia, Books, ArXiv, StackEx-change, TruthfulQA, ToxiGen	Text data	Efficient, high-performance open model	Self-Supervised, RLHF	[43]
Surface Normal Estimation, Depth Estimation, Semantic Segmentation, Edge Detection	UVD-v1, Kinetics-700, ImageNet	Image/Video	Unified, multi-task visual learner	Self-Supervised	[44]
Self-Supervised Universal Visual Representation	LVD-142M, ImageNet-1K, ImageNet-A, ADE-20K	Image data	Scalable state-of-the-art vision encoder	Vision Trans- former (ViT) trained with self- supervision	[45]

summarizes some of the recent developments in foundation models for various tasks in different domains.

3.1. Large-Scale Data Preprocessing and Integration

Foundation model development heavily depends on integrating diverse data sources. To train them effectively, massive datasets are gathered from various sources, including text, images, and videos. This data is then preprocessed and cleaned to ensure uniformity and high quality, enabling foundation models to learn from vast, multi-domain data. This ultimately improves the capacity for generalization of the foundation models by subjecting them to a variety of patterns and variances in the data. Domain-specific preprocessing entails adjusting data preparation techniques such as image augmentation in vision and tokenization in NLP, to the requirements of particular domains, ensuring that foundation models can perform optimally in specialized contexts. It also increases the robustness and generalization capabilities of the models across various domains [9,46] and promotes high performance for specific applications, such as medical video analysis. Endo-FM, a large-scale foundation model, is an example of a foundation model that relies on large datasets that combine different sources. Additionally, different cross-domain knowledge transfer methods are used for effectively transferring knowledge from one domain or task to another with minimal retraining. It is essential for applications that require cross-sectoral knowledge that is flexible. Text, image, and audio integration techniques assist models like GPT-4 and CLIP to handle multiple data types simultaneously, allowing for rich, cross-modal understanding and generation [47,48,49].

3.2. Advanced Training Architecture and Techniques

Foundation models employ transformer architectures that utilize attention mechanisms to identify relationships across lengthy data sequences like text while also enabling efficient handling of different data types like images and audio within a single model. These models can capture dependencies and relationships over a large context because of techniques like long-range and hierarchical attention, which enhance performance on tasks requiring lengthy or complex input sequences, including document summarization and video analysis [50]. These models can perform exceptionally well in both language generation and understanding, e.g., GPT and BERT [51,52]. Self-supervised learning is a fundamental technique of foundation models, enabling them to learn from vast amounts of unlabeled data, mimicking the way that humans acquire knowledge without explicit instruction. According to Jakubik et al., “foundation models are pretrained on a large volume of unlabeled data to learn representations without explicit guidance from the ground truth” [53]. Large foundation models are developed using billions or even trillions of parameters, demonstrating high-parameter scaling. Because of their high parameter count, they can capture complex and subtle relationships within large datasets, which enables them to provide comprehensive understanding across various applications. Recent developments related to self-supervised learning include the following:

• Advanced techniques for learning representations by comparing and contrasting similar and dissimilar samples: This technique is used to enhance the capability of the model to distinguish between similar and dissimilar instances, which is essential for applications like text embeddings and object identification [54].
• Masked language modeling: Masked language modeling (MLM), which is commonly used in models like BERT, involves randomly masking parts of the input and then training the model to predict the missing pieces. These are basically complex methods for predicting the masked input data segments towards improved model robustness and interpretations, which aid the model in achieving a thorough understanding of dependencies and data structures [55].

3.3. Efficient Fine-Tuning and Adaptation Methods

Task-specific fine-tuning is done on the models after pretraining with smaller datasets with labels for specific tasks, elevating their performance and computational efficiency. This makes foundation models versatile and flexible in tackling particular use cases [56].

Efficient Fine-Tuning (PEFT): An advanced method for adapting models with minimal parameter updates. Techniques like Adapters or bottleneck layers allow fine-tuning with minimal parameter adjustments at the cost of efficiency while preserving flexibility [57].

Low-Rank Adaptation (LoRA): These are methods for efficient adaptation through low-rank matrix decomposition. LoRA decomposes the model parameters into low-rank matrices for resource-efficient adaptation with minimal memory overhead, which is well-suited for smaller-scale applications [58].

Prompt Engineering: It is the process of carefully constructing input prompts to guide the model’s responses without changing the weights of the base model. For tasks such as dialogue generation or question answering, carefully crafted input prompts can guide the model’s responses effectively without altering model weights, especially useful for NLP applications [59].

Fine-tuning techniques include the following: (i) supervised fine-tuning, which adapts the model’s knowledge to specific tasks, e.g., classification and translation. This approach preserves general knowledge from pretraining while refining skills for specific applications. (ii) Meta- learning algorithms also improve the model’s adaptability by optimizing it to quickly learn new tasks from limited data, enhancing its efficiency in dynamic settings. (iii) Architecture optimization involves modifying the model’s architecture such as adjusting its layer configurations or adding specialized modules to optimize performance on specialized tasks like object detection or language translations.

3.4. Multimodal Integration Capabilities

More and more foundation models are used in various domains that use various data types like text, images, or video, executing tasks such as translation or image creation [20,56]. They are capable of understanding various datatypes and creating unified representations of data from different sources.

• Unified Representation Spaces: By projecting the different modalities to a common space, models like CLIP and GPT-4 create unified representation spaces to process and align different data types. These multimodal models are able to extract and relate information coming from diverse sources and jointly analyze it, like images and text, for applications in image captioning or video–text alignment. This enables foundation models to apply knowledge from one modality to another effectively [9,60].
• Modality-Specific Attention: The attention mechanisms are adapted to serve various types of data in an integrated model so that the system can analyze and highlight information according to the peculiarities of each modality, such as temporal attention for videos or visual attention for images, to capture modality-specific information more accurately [61,62].
• Cross-Modal Alignment: Techniques aligning representations across modalities ensure that, for instance, the word “dog” in the text is aligned to a picture of a dog. Such alignment becomes indispensable in tasks like image-text retrieval and multimodal translation. In the coming section, we will discuss the applications of foundation models across various domains in detail [63,64].

3.5. Robust Infrastructure and Training Platform

Training of foundation models requires specialized hardware like GPUs and TPUs to handle massive datasets and large architectures of the models. The training process of the model may take days or even weeks, depending on the size and complexity of the model. Platforms like Amazon Sage Maker, Google Vertex AI, and Microsoft Azure AI provide scalable environments for data preparation, model building, and deployment, which are essential for efficient training and serving large foundation models [65]. Amazon Sage Maker provides an end-to-end machine learning infrastructure where organizations can prepare data, build models, and deploy them in an efficient manner. IBM Watson provides enterprise-grade AI model development capabilities with specialized tools for various industries. Google Cloud Vertex AI provides unified ML model development with advanced AutoML capabilities. Microsoft Azure AI provides the scalable infrastructure required for training large-scale AI models and deploying them, supporting various frameworks and tools. A company could use one of these models, such as the Hugging Face model, train it using its own proprietary data, and then fine-tune it using prompt engineering to perform specific tasks [66,67].

3.6. Safety, Security, and Alignment Techniques

Alignment techniques are a set of strategies that align foundation models with ethical guidelines so that they conform to pre-defined safety protocols, which are imperative in scaling up the models. Additionally, when models are deployed into diverse, real-world environments, robust techniques allow them to generalize well to new distributions of data so that good performance and reliability are preserved.

3.7. Continual Assessment and Interpretability

Base models, after training, are thoroughly evaluated on benchmark datasets to quantify performance and ensure compatibility for intended applications. If the outcome is not satisfactory, the model might be further improved by modifying hyperparameters or even returning to the data preparation phase for quality enhancement. The evaluation also includes model interpretability techniques to help explain to stakeholders the decision processes used by the model. While more complex models demand techniques for interpretability, such as attention visualizations and feature attribution analysis, to provide users with insight into the model’s predictions and how decisions are made.

3.8. Types and Examples of Foundation Models

Foundation models vary across many dimensions—architecture, training objectives, and applications—all for exploiting various aspects of learning and interaction with data. We shall explain below in detail various types and examples of foundation models, as shown in Figure 9.

• Autoregressive Models

While autoregressive models, which include the GPT series, like GPT-2, GPT-3, GPT-4, and XLNet, are usually trained to predict the next word in a sequence given all the previous words, this kind of training allows these models to produce coherent and contextually relevant text, which is very helpful in creative writing, chatbots, and personalized customer service interactions.

2.

Autoencoding Models

All autoencoding models, such as BERT and RoBERTa, are usually trained to understand and reconstruct their input after corrupting it—usually through a process called masked language modeling, in which the model, during training, is deprived of random tokens. This means the model learns how to predict missing words given only their context. This makes them quite useful for understanding the structure of the language and applications such as text classification, entity recognition, and question answering.

3.

Encoder–Decoder Models

T5 and BART are encoder–decoder models with special capacity for turning input texts into their output versions. These models are among the most capable ones in performing such difficult tasks as summarization, translation, or text editing since they learn to encode an input sequence into a latent space and then decode it into an output sequence. Since they are usually trained on different kinds of text-to-text conversion tasks, they can be applied in almost any domain.

4.

Multimodal Models

Multimodal models, such as CLIP by OpenAI and DALL-E, process and generate text and images along with other types of data. The more this kind of multimodal information a model understands and generates, the better it is at handling tasks that are inherently focused on interpreting the correspondence between images and their textual descriptions in image captioning, finding images according to a text description, and vice versa.

5.

Retrieval-Augmented Models

RETRO: Retrieval-Enhanced Transformer is a series of models that extend the power of traditional language models with external knowledge retrieval. This architecture design allows the model to retrieve specific information at prediction time from a large database or corpus and fold it into the output, vastly improving performance on tasks that require a high degree of factual accuracy and detail, such as question answering and content verification.

6.

Sequence-to-Sequence Models

Such models, like Google’s transformer and Facebook’s BART, fall under a class of models known as sequence-to-sequence models. These perform tasks where an input sequence is to be transformed into another, usually related output sequence. These models are the backbone of machine translation and document summarization, where either the entire content or meaning needs to be captured accurately and expressed in another form.

3.9. Examples of Foundation Models

Each of these foundation model types is uniquely suited to different tasks because of its peculiar way of training and operational design. Foundation models can be either generative or non-generative based on the ways they perform.

Generative models generate new content based on input: text, images, music, etc. These are systems that, instead of simply processing data, can actually generate new data. Non-generative models normally deal with tasks related to classification, analysis, or even understanding of the data without any generation in between—for example, text understanding or processing an image.

As stated earlier, foundation models are usually trained to gather valuable general information from the data rather than performing any specific tasks. Nowadays, foundation model-based products are used by millions of people across the world. Foundation models, which were primarily developed for text generation and information retrieval purposes, have now been broadened to incorporate different types of inputs such as images, videos, and sounds [9] and are able to perform many tasks.

• Autoregressive LLMs are able to guess the subsequent words in a sentence and generate content; for example, GPT3, Chinchilla, PaLM, etc. [68,69].
• Text-to-image models are used to understand and generate images based on text prompts. They can be fine-tuned to better suit human interests or to carry out some specific tasks; for example, DALL.E [70].
• Different types of contents can be generated using foundation models; for example, GPT-3 is used for text, DALL.E for images, and Codex for code [71,72].
• Additionally, BERT and CLIP are some of the foundation models that can be employed for non-generative tasks, like classifying and representing text differently, giving a one-value output instead of a lengthy unstructured one, or estimating a numerical value from an image or visuals [72,73,74].
• A lot of tasks fall between the spectrum of either generative or non-generative, and the majority of the foundation models can be fined-tuned or adapted to function as either.
• Open AI has developed a versatile chatbot named ChatGPT, which reached an estimate of more than 100 million daily active users.
• Another foundation model, Midjourney, is deployed to generate millions of images from text prompts per day. As more and more foundation models are being integrated into the products, their access to broad audiences is also increasing. A number of famous companies are preparing to introduce products like ChatGPT; for instance, Meta’s LLaMa, Jasper AI, Microsoft’s Bing Chat, etc.

  Table 7. Summary of foundation models and their applications across domains.

  Foundation Model	Company/Develop	Domain Category	Specific Domains	Key Features and Applications
  LaMDA [75]	Google	Multi-domain NLP	Conversation, General Knowledge	Experimental AI chat service, natural dialog, conversational capabilities
  GPT-3.5 [76]	OpenAI	Multi-domain NLP	Conversation, Text Generation, Analysis	Human-like conversations, general- purpose dialogue
  GPT-4 [77]	OpenAI	Multi-domain NLP	Text Generation, Visual Understanding, Coding, Analysis	Powers multiple AI assistants, enhanced coding capabilities, DataLab user support
  Codex [78]	OpenAI	Specialized NLP	Code Generation, Development Tools	Real-time code suggestions, function completion, programming assistance
  AudioLM [79]	Google	Audio Generation	Music Creation, Audio Processing	Text-to-music generation, music composition
  BLOOM [80]	Hugging Face	Multilingual NLP	Multilingual Processing, Code Generation, Multiple NLP Tasks	46 language support, 13 programming languages, direct model access
  LLaMA [81]	Meta	Research NLP	AI Research, Model Development	Research advancement, foundation model development
  DALL-E 2 [82]	OpenAI	Computer Vision	Image Generation, Art Creation, Design	Text-to-image generation, realistic art Creation
  BERT [83]	Google	NLP	Text Analysis, Document Processing, Question Answering	Bidirectional model, 3.3B token training, biomedical applications
  Cohere [84]	Cohere	Specialized NLP	Text Generation, Enterprise Solutions	Specialized tasks, enterprise solutions
  Stable Diffusion [85]	Stability AI	Computer Vision	Image Generation, Art Creation	Resource efficient, consumer hardware Compatible
  Gato [86]	Google Deep- Mind	Multimodal AI	Robotics, Gaming, Vision	Multimodal system, robotics integration, multiple task capabilities
  PaLM-E [87]	Google	Robotics and Vision- Language	Robot Control, Visual Understanding	Cross-domain integration, knowledge transfer
  Action Transformer (ACT-1) [88]	Adept	Human–Computer Interaction	Web Navigation, Tool Us- age	Browser automation, digital tool interaction
  Midjourney [89]	Midjourney	Computer Vision	Image Generation, Art Creation	Advanced image generation, photorealistic output
  Spark [90]	Meta	Computer Vision	Image Filtering, Social Media, Image Augmentation	Instagram filter generation, image style Augmentation
  Chinchilla [91]	DeepMind	Multi-domain NLP	Text Generation	Content creation
  PaLM [39]	Google	Multi-domain NLP	Text Generation	Content creation
  CLIP [92]	OpenAI	Computer Vision and NLP	Image and Text Classification	Image tagging, cross-modal understanding

  summarizes some examples of foundation models and their applications across various domains.

4. Application of Foundation Models

The following areas could benefit significantly from the many options and use cases that foundation models could offer because of their broad adaptability.

4.1. Natural Language Processing

Foundation models represent the backbone for many different applications based on NLP. Most of the foundation models in NLP are pretrained on a very large corpus with various pretraining tasks, equipping them with language understanding and generation capabilities. Indeed, it is the thorough pretraining that enables fine-tuning and makes them easily adaptable to various types of downstream tasks. They are used in machine translation to offer seamless communication across various languages [93]. They can also be applied to tasks such as sentiment analysis—understanding emotional tone—or even chatbot development for more natural human–computer interaction. Language is a fundamental tool for human communication, thought, relationships, knowledge sharing, and social development. Languages may vary from culture to culture, but they demonstrate some common features to enable effective human interaction [94]. Language generation and understanding are essential for AI nowadays. Natural Language Processing (NLP) focuses on enabling computers to comprehend and produce human language along with its relevant fields: text-to-speech and automatic speech recognition (ASR). Previously, NLP was focused on creating solutions for complex tasks, with the goal of developing models that function effectively in subsequent applications. Among these tasks are sentiment classification or analysis (determining whether a review is positive or negative based on words and sentences), labeling of sequences (identifying names of objects or parts of speech), classification of span relations, and generation tasks like speech recognition, summarization, and translations [95]. However, in 2018 and 2019, with the advent of foundation models like ELMo [96] and BERT [83], the emphasis of NLP has changed to understanding and applying these foundation models to perform various tasks. Foundation models have transformed the field of NLP; the field now considers foundation models as primary tools for achieving broader language acquisition and learning. As stated earlier, they are pretrained on a very large amount of text data, which aids them in performing a variety of language-related tasks with significant accuracy. Some examples are as follows:

• Text Generation: FMs like GPT-3 and ChatGPT have been used to generate coherent, contextually relevant text; hence, they enrich applications in content creation and conversation agents [12].
• Text Classification: They are exceptionally good at categorizing text data, which is highly important in sentiment analysis and information retrieval [97].
• Masked Language Modeling: This is the popularly used pretraining task in NLP, where many models, like BERT and SpanBERT, are trained based on the objective of predicting randomly masked words in an input sequence. It helps the model to learn contextual representation, which is very important for most tasks like sentiment analysis, question answering, and named entity recognition [98,99].
• Denoising AutoEncoder (DAE): Models such as BART [100] make use of DAE tasks, which involve adding noise to input text and then making the model learn to reconstruct the original, clean text. These help in effectively carrying out tasks that involve text generation, such as summarization or paraphrasing, since this enhances the model’s robustness against incomplete or noisy data.
• Replaced Token Detection (RTD): ELECTRA comes up with RTD as a discriminative task, where it learns whether a token has been replaced by another. This in turn helps in gaining insight into deep language understanding and proves handy for sentence classification and language inference tasks [101,102].
• Next Sentence Prediction: BERT is used for NSP tasks to predict whether two sentences are connected logically. This enhances a model’s ability in sentence coherence tasks such as document classification and natural language inference.
• Sentence Order Prediction: ALBERT is used where the model has to predict the correct order of sentences. The pretraining based on SOP enhances the discourse structure understanding capability of the model and thus benefits long-form question answering and summarization tasks [103].

These will surely help FMs learn robust language representations, which can then be fine-tuned for various applications, achieving state-of-the-art performance on diverse NLP tasks.

4.2. Advancing Computer Vision

Foundation models have significantly advanced the field of computer vision by pro- viding robust, scalable solutions to different applications. These models, pretrained on extensive datasets, show outstanding performance in both generative and discriminative tasks, making them a very versatile tool in the domain. FMs can execute multiple tasks in image and video analysis. They are also especially valuable in application domains like Earth observation and medical imaging, where labeled data can be scarce. Vision is one of the primary human senses, enabling us to observe and understand the environment around us. It aids us in gathering information about our environment, identifying objects, and navigating our surroundings. Despite being a simple task for humans, transmitting visual perception and observation abilities to machines has proven to be an extremely challenging task, called “AI paradox” by Hans Moravec. Computer vision is a field of AI that intends to enable machines to interpret and understand the visual world the same way humans do. It entails developing algorithms and methods that allow computers to process and evaluate visual data from images and videos. While it is a complex process, researchers have still managed to develop algorithms that are able to figure out scenes, identify objects, and classify images. With the development of ImageNet, computer vision has been transformed greatly from task-specific feature engineering to a broader and more generalized approach [92,104]. The outcomes for tasks like image recognition, object detection, and image segmentation have been greatly improved by first training models on huge datasets and then fine-tuning them for multiple tasks. This concept became the core principle for the development of foundation models. These traditional supervised techniques were, however, really expensive and heavily relied on carefully collected labeled data to perform well, which ultimately limits their applicability and generalization. On the other hand, foundation models show major advancements in the field of computer vision by employing self-supervised learning and extensive unannotated data, promising new scalability, generalization, and multimodal integration potentials [9,105].

Key applications of the foundation model in computer vision will be presented here.

• Medical Imaging: FMs can be applied for medical image analysis to support the doctor in diagnosing different diseases. They can also generate realistic images from textual descriptions and other images.
• Object detection and tracking: FMs make it possible to locate and identify objects in images or a video frame and label them appropriately, which is useful for various tasks. For example, in security systems, they are used for object identification and detection in surveillance footage. Vision transformer embeddings from the foundation models have been applied to single object tracking and have outperformed the classical models, such as ResNet-50, with exceptional results as indicated [106].
• Visual Place Recognition: According to a study by [107], foundation models, especially DINOv2, demonstrate striking advantages in the visual place recognition task and have even reached state-of-the-art status under challenging conditions, including occlusion and seasonal changes. Self-attention features promote re-ranking towards improved location prediction accuracies for geotagged images.
• Histopathology: Foundation models have been benchmarked as feature extractors and for generalizability in tasks like slide-level and patch-level classification within computational pathology, where it has been shown that low-resource fine-tuning achieves and sometimes outperforms the state of the art on image classification tasks [108].
• Enhanced localization: The FM-Loc approach uses CLIP for object detection and GPT-3 for semantic labeling, creating a very robust image descriptor invariant to changes in scene geometry and camera viewpoints. It has proven to be effective for several indoor scenes without extensive training or fine-tuning, thus demonstrating that FMs are adaptable [109].
• Creative design: Models learn to predict missing parts of images, aiding object detection and image segmentation, and also designing new products of artwork. They are also used for creating realistic special effects in movies and videos.

4.3. Graph Learning

Large language models have been increasingly integrated into graph learning, enhancing applications including recommendation systems and social network analysis. In this respect, it will be easier to develop graph foundation models that employ both the strengths of large language models and those of graph neural networks, ensuring superior performance on a wide range of graph-based tasks. The following sections outline some key applications and improvements in this area.

• Graph Learning with LLMs: There will be some interesting applications in the sense that LLMs can generate high-quality textual features of graph entities and edges, which enhance graphs to be used in graph learning tasks, or they might encode knowledge that can enhance graph data toward a better understanding and analysis of complex relationships [110,111].
• Graph Foundation Models Development: GFMs are designed to generalize across a wide range of graph data, addressing challenges in zero-shot learning settings. In particular, these models have used unified graph tokenizers and scalable graph transformers to capture node dependencies effectively [112].
• Graph Classification: FMs are also employed for graph learning tasks, which provides evidence of their adaptability across different data modalities [113].

4.4. Transforming Healthcare

The foundation models hold much potential for the transformation of healthcare and biomedicine. They can be fine-tuned for various tasks, from the personalized care of a patient to biomedical research, by unifying diverse sources of medical knowledge and updating them.

• Pathology: Applications of FMs in pathology come in the form of disease diagnosis, diagnosis of rare cancers, prediction of patient survival prognosis, biomarker expression predictions, and immunohistochemistry expression intensity scoring, which indeed proves the applications to be versatile with high accuracy in the health care domain [114].
• Mental Health: They are applied to electronic medical records for the prediction of diseases and personalized treatment. Large language models support therapy and monitoring of a patient, and pharmacogenetics deals with precision medicine—all for enhanced mental health care through integrated data sources [115].
• Health Professional Support: Tasks such as HERs (electronic health record) management, summarization of history, and suggestion of lab tests and treatments will minimize administrative inefficiencies and avoidable medical errors to further raise the quality of care at lower costs [116,117,118].
• Improved Patient Interface: Applications that directly face the patient can facilitate health literacy and engagement for the patients by responding to questions, explaining clinical facts, communicating on population health, and much more, including emergency precautions like COVID-19. These types of models will improve the health care service and health literacy of patients, acting as a trustable and reliable source [119,120].
• Better Efficiency in Public Health: The fact that the foundation model contributes to faster diagnosis and prioritization of treatments during pandemics translates to utilizing resources at the right time to manage or prevent further spread. This is well-documented in the works of [121].
• Accelerated Drug Discovery: These models identify drug targets, design molecules, and optimize protocols far more quickly and inexpensively than what was ever achieved before in this field [122,123]. An IBM foundation model, CogMol (Controlled Generation Molecules), has recently produced a series of novel COVID-19 antivirals by using a common architecture known as variational autoencoder.
• Precision Medicine: Integrating data from multimodal sources, foundation models will be able to offer personalized treatments to individual patients, allowing for more precise care—through the combination of data in systems like EHRs and genetics [124,125].
• Smarter Clinical Trials: Foundation models will make clinical trials smarter by predicting outcomes, designing eligibility criteria, and automating patient matching; therefore, trial success rates can be raised, and costs can be reduced [126,127].

4.5. Law

The foundation models can further cut down some of the procedural and financial barriers to accessing justice. Most legal tasks involve specialized languages and a general scarcity of labeled data, which are challenges to which few-shot learning of foundation models is well-suited.

• Public law: FMs could support clients and attorneys in civil law matters by reviewing contracts, patent searching, and legal document translation [128,129]. Models can also identify, at low cost, the relevant legal issues from client descriptions, which is an area where legal accessibility is gaining momentum [130].
• Criminal Justice Assistance: FMs can help public defenders handling caseloads in criminal justice become more efficient, reducing erroneous procedures. This may also include eliminating bias through standardization of the processes in charging decisions and identification of discriminatory language use in police records [131].
• Improvement of Public Law: Foundation models can improve the quality of services and efficiency delivered from governments, supporting everything from the process of public comments to assisting patent examiners. For example, in regulatory review, foundation models may also improve NLP applications and Freedom of Information Act requests [132].

4.6. Education

Foundation models hold the key to making education inclusive and available to everyone. While education has been identified as a priority area in societal development, large-scale access to quality education still remains an elusive dream because of economic and logistical hurdles.

• Ethical Considerations: Bias in data and the aspect of privacy are two major concerns that must be addressed for equity in accessing educational opportunities [133,134].
• Understanding: Foundation models can help diagnose students’ misconceptions and provide targeted feedback. The knowledge tracing in MathBERT and GPT-powered feedback on open-ended tasks are examples of how this could be done [135,136].
• AI for Instruction: These models have the potential to personalize instruction with regard to content adaptation and pedagogical techniques. Such sources can include training resources like question-and-answer forums such as StackOverflow, or even Wikipedia, to train AI in tutor-like activities, while public data on lecture videos or textbooks could inform models in their effective communication with students [137,138].

4.7. Telecommunication

Foundation models have begun to transform the telecommunications industry with new ways of network management and optimization. They reveal an extraordinary ability to generalize across very diverse network deployment scenarios, from urban centers to remote locations, allowing the creation of more adaptive and efficient network operations. Most importantly, they are precious for lessening the traditional reliance in the industry on large, labeled datasets; in other words, they make network optimization more accessible and less expensive. In addition, FMs can be applied in various aspects of telecommunication operations, such as network traffic prediction, anomaly detection, spectrum management, and optimization of quality of service. The ability to learn from unlabeled data and knowledge transfer between different network contexts further makes them very helpful in dealing with increased complexity in modern telecommunications infrastructure, especially in the era of ongoing deployment of 5G and preparation for 6G networks [139].

4.8. Industrial Applications

Foundation models have the potential to transform industrial processes through their advanced NLP capabilities and complex reasoning abilities. Applications in process optimization, quality control, supply chain management, and predictive maintenance can all be transformed. They can analyze vast amounts of unstructured data, identify patterns, and make sophisticated decisions that were previously impossible or highly resource-intensive [140]. The integration of FMs could bring about substantial improvements in industrial efficiency, decision-making processes, and the ability to innovate, which implies a promising but currently underutilized frontier in industrial digital transformation.

4.9. Geophysical Research

Applications of FMs in geophysics, therefore, represent a massive stride in the analysis and interpretation of Earth science data. In fact, FMs are very important for processing large amounts of complex geophysical data from exploration surveys and remote sensing operations. Such an ability to handle large datasets and incorporate multimodal information, such as seismic data, satellite imagery, and ground-based measurements, addresses long-standing challenges in the field of geophysics itself. FMs can be used in various geophysical applications, including subsurface mapping and mineral exploration, natural hazard assessment, and climate studies. This capability of the models to learn patterns across diverse types of data helps geoscientists extract meaningful insights from datasets that continue to grow in size and complexity, which may change our understanding of Earth’s systems and processes forever [141].

4.10. Earth Observation and Geospatial AI

Foundation models are excellent for various land cover classifications, crop-type mapping, flood segmentation, and many other tasks. They are far better compared to problem-specific models when labeled data is small [142]. Foundation models are also very effective at jointly solving multiple problems. They show better performance in Geospatial AI. Prithvi is a transformer-based foundation model that was trained on 1 TB of satellite data and achieves state-of-the-art results for tasks such as cloud gap imputation and flood mapping, thereby illustrating the power of foundation models for high-accuracy and efficient geospatial AI applications using limited labeled data [53].

4.11. Revolutionizing Robotics

Robot learning is being revolutionized by foundation models (FMs). Researchers are creating strong models that can learn from enormous volumes of data and generalize new tasks by utilizing simulators, datasets, and frameworks. These models show great promise in tackling issues like generalization performance and dynamic data, which are critical for robots functioning in real-world settings. They are also increasingly used with robots to improve the manipulation, navigation, and reasoning capabilities of robots [143]. Foundation models can help develop versatile robots that can perform various tasks by learning guidelines that comprehend task descriptions, such as “make breakfast” which may indicate different actions for different users (e.g., cooking toast for one user vs. preparing pancakes for another) [144,145]. They can also be used to translate task descriptions or details assigned by humans (verbal commands) into machine-understandable formats like tangible, quantitative reward signals that evaluate task performance and completion. This could be achieved by using various modalities, such as natural language, videos, and feedback interaction [146,147,148]. Such robots can be trained on various multimodal datasets, which enable them to fit into new environments and tasks like cooking in distinct kitchen setups and layouts. Assumptions from these large datasets can be utilized by the model to achieve generalization effectively, even with limited samples [149,150,151]. In closed-loop situations where actions directly affect perception, foundation model could assist in dynamics modeling, policy learning, and inverse reinforcement learning, all of which are very important and essential for robots to learn to behave [152,153,154]. Robotic data contains synchronized, multimodal sensor streams. Robots can learn action-conditioned dynamics and policies using existing data resources like simulations, autonomous robot interactions, texts, and videos of human performing different activities [155,156], in order to learn generalizable skills, which reduces the gap between simulated and real-world environments for accurate robot learning [157,158]. Foundation models are also finding their increasing application in robotics, especially for tasks in unstructured environments, where traditional models often fall short. These include large language models and visual representations that help robots to perceive, make decisions, and control the working environment with much more adaptability and intelligence.

Applications in Unstructured Environments

• Generalization: Foundation models exhibit superior generalization, allowing robots to tackle tasks not explicitly present in training data, which is crucial in unpredictable settings like construction sites [159,160].
• Task-Specific Policies: The Diffusion for Policy Parameters (DPP) approach enables the creation of modular, task-specific policies, enhancing interpretability and user interaction [161].
• Object-Centric Representations: The POCR framework combines “what” and “where” information from pretrained models, improving performance in robotic manipulation tasks [162].

5. Challenges

Challenges of Foundation Models by Domain

5.1. Healthcare

• Ethical Data Acquisition: Some modalities, like CT, though valuable for diagnostics, pose health risks. Collecting such data solely for training violates ethical norms, limiting proactive dataset creation [163,164].
• Privacy and Governance: Healthcare data, especially genetic data, faces stringent restrictions due to privacy laws. Poor data governance during collection or training can lead to sensitive information leakage and misuse, raising ethical and legal concerns [163,164,165,166].
• Dataset Limitations: Most medical datasets are imbalanced among different imaging modalities, such as PET and DSA, and among pathologies. The rarity of some diseases or high cost of imaging makes the situation worse, limiting generalization to wider clinical applications [167,168,169,170]. In addition to imbalance, recent research has found that apparently random or heuristic sampling strategies apply when collecting and preprocessing data to overcome the imbalance issue. However, such strategies still struggle with biasness [171].
• Data Heterogeneity: Variations in patient populations, medical centers, and protocols result in mismatched data distributions. Moreover, concept drift due to the evolution of medical practices and knowledge adds complexity to long-term data reliability [172,173,174,175].
• High Annotation Cost: Specialized imaging, such as CT, is very costly. Moreover, annotation of datasets requires qualified personnel, making the overall process very expensive [169,176,177,178].

5.2. Natural Language Processing (NLP)

• Bias Amplification: Models acquire and amplify biases present in training data, which has been associated with causing societal harm by reinforcing stereotypes—for example, gendered professions [179,180,181].
• Propagation of Harmful Language: They can be mistakenly used to generate overtly abusive or defamatory content, perpetuating hate speech or misinformation [180,182].
• Privacy Exposure: Large models may leak any personally identifiable information retained from training texts unwittingly [183].
• Apparent Fluency Risks: While the produced output usually appears fluent and coherently smooth, it can embed incorrect or biased information while deceiving users [184].
• Maliciousness: Foundational models give rise to extremist content generation and misinformation in critical realms like political propaganda or false news [182].

5.3. Computer Vision

Challenges in Computer Vision and Foundation Models

• Evaluation and Benchmarking: Vision-language models are difficult to evaluate due to their reliance on external models (e.g., GPT-4) for judging tasks, which often overlook visual-specific nuances [185].
• Hallucination: Models sometimes generate irrelevant or incorrect outputs by over-relying on text input while ignoring visual data [186].
• Bias and Fairness: Training data biases can lead to skewed results, such as models misclassifying images or perpetuating stereotypes [187].
• Vulnerability to Adversarial Attacks: Vision models are susceptible to attacks where small corruptions in inputs produce incorrect outputs, threatening their reliability [188,189].
• High Compute Requirements: Large vision models have high resource requirements, rendering real-time applications such as segmentation infeasible [186].

5.4. Robotics

• Computational Constraints: Foundation models need substantial amounts of computation, which renders them difficult to deploy on robots with limited hardware, especially for real-time tasks [190].
• Modality Integration: Combining multiple data types like 3D point clouds and text is difficult due to the lack of aligned datasets and the need for intermediate data conversions, which may cause information loss [191].
• Uncertainty Quantification: Robotics requires reliable uncertainty estimates for safety-critical tasks, but models often struggle with uncalibrated or incomplete estimates [192].
• Sim-to-Real Gap: Simulators are not able to generate the same level of variation and realism as real environments, which often causes shifts in performance at deployment [193,194].
• Platform and Environment Variability: Platforms vary in their sensors and abilities, and environments vary broadly, making generalization across uses difficult [195,196].
• Safety Evaluation: Failures are to be found through extensive pre-deployment testing and through robust runtime monitoring; however, ensuring this is the case remains a challenge [197,198].

5.5. General

• Environmental Impact: Most of the training and usage of foundation models have enormous energy requirements, leading to massive carbon emissions.
• Bias and Representation: Hegemonic positions in big datasets encode, diffuse, and harm marginalized communities through the consolidation of stereotypes [180,199].
• Scalability and Cost: With financial scaling coming at a high cost for large models, there are greater disincentives for resource-poor researchers and damping of innovation.
• Interpretability: Foundation models’ “black box” decisions cannot be comprehended or trusted for accountability.
• Dual-Use Risks: These models can be misused to create harmful content, such as extremist propaganda or deepfake materials [182].

5.6. Ethical Implications

FMs that are trained on big, heterogeneous data are prone to bias and unfair representation, which stems out of societal disparities found in the source data. This is particularly vital in such a field as healthcare, where biased predictions can imbalance underrepresented groups. During pretraining or model deployment, privacy issues are also possible because sensitive information is accidentally revealed [11]. The misuse of FMs in surveillance, generation of misinformation, or automated decision-making increases the level of ethical stakes.

5.7. Environmental and Societal Considerations

Foundation models are pretrained and fine-tuned using massive computational resources, which consume large amounts of energy and emit carbon, and thus have sustainability implications [193]. In addition to environmental expenses, their implementation may also have social effects, such as the spread of misinformation, the sensationalization of biased or detrimental content, and the robotization of the labor market. To overcome these concerns, it is essential to use energy-efficient methods of training, responsible management, and close supervision of the downstream implementations to implement sustainable and socially responsible AI applications [11].

6. Future Prospects

Foundation models have taken a transformative role in many fields since the ability to pretrain on extensive datasets and adapt to a wide array of downstream tasks can be found within them. The robustness they show with generalization and multimodal integration opens paths of innovation and efficiency, with promising applications in healthcare, robotics, natural language processing, and computer vision. These models unlock breakthroughs through an overall reduction of training on task-specific data and also significantly facilitate cross-domain advancement. FMs are used to enhance diagnostic precision, enable personalized treatment plans, and facilitate seamless data sharing while preserving patient privacy for overall efficiency and outcomes. The capabilities of FMs, like ChatGPT in healthcare, range from decision support to summarization and interpretation of medical reports, thus benefiting patient care and access [200]. Similarly, in bioinformatics, FMs are set to revolutionize sequence analysis, structure prediction, and multimodal integration, addressing key challenges in computational biology that promote innovation. Robotics has also benefited from these models, with applications in robot learning enabling advancements in manipulation, navigation, and planning through generalizable AI capabilities [143]. In the case of autonomous driving, these models enhance safety through better scene understanding, rare-event data augmentation, and enhanced predictive capabilities that will make the systems much safer and wiser. Similarly, in NLP, FMs provide enhanced context- aware responses, which can transform customer service and automated reporting applications. In computer vision, FMs offer improvements in visual understanding, driving progress in autonomous vehicles and surveillance systems. Despite their huge potential, most of the challenges faced by FMs include data scarcity, explainability, and the need for safety guarantees within critical applications. Other challenges also include predictive uncertainty, high computational demands, ethical issues related to privacy and fairness, and inaccuracies in data that maintain their vital status, especially in healthcare and other safety-critical areas. On top of that, there is a risk of homogenization and a power struggle among corporations; it would be important to have equitable access, transparency, and responsible development in order for them to realize their full transformative impact. Therefore, future studies must focus on these challenges by nurturing robust mechanisms of model validation, mitigating bias, and setting ethical guidelines. Efforts in this direction are crucial for realizing the full potential of foundation models and ensuring responsible deployment across sectors [201,202,203].

7. Conclusions

The field of foundation models (FM) is growing fast, and it is being used significantly across various fields. Our quantitative analysis demonstrates that the total number of papers related to FM has increased at an unprecedented rate, from fewer than 500 publications in 2020 to over 9000 papers by 2025, making it one of the fastest-growing areas of AI research. The highest number of papers on foundation models was published in the year 2024 so far. The strengths of FMs, in particular, multimodal reasoning, transferability, and task generalization, are also promoted in large-data domains. Nonetheless, the experiences in healthcare and safety-related domains indicate continuing constraints, such as the lack of domain adaptation measures, limited benchmarking datasets, and unreliable documentation of assessment criteria. The most effective systems, in terms of methodology, are based on large-scale self-supervised systems, mixture-of-experts systems, and effective fine-tuning strategies, but reproducibility is still not consistent.