Artificial Intelligence in Healthcare: How to Develop and Implement Safe, Ethical and Trustworthy AI Systems

Abstract

Background/Objectives: Artificial intelligence (AI) is increasingly integrated into everyday life, including the complex and highly regulated healthcare sector. Given healthcare’s essential role in safeguarding human life and well-being, AI deployment requires careful oversight to ensure safety, effectiveness, and ethical compliance. This paper aims to examine the current regulatory landscapes governing AI in healthcare, particularly in the European Union (EU) and the United States (USA), and to propose practical tools to support the responsible development and implementation of AI systems. Methods: The study reviews key regulatory frameworks, ethical guidelines, and expert recommendations from international bodies, professional associations, and governmental institutions in the EU and USA. Based on this analysis, the paper develops structured questionnaires tailored for AI developers and implementers to help operationalize regulatory and ethical expectations. Results: The proposed questionnaires address critical gaps in existing frameworks by providing actionable, lifecycle-oriented tools that span AI development, deployment, and clinical use. These instruments support compliance and ethical integrity while promoting transparency and accountability. Conclusions: The structured questionnaires can serve as practical tools for health technology assessments, public procurement, accreditation processes, and training initiatives. By aligning AI system design with regulatory and ethical standards, they contribute to building trustworthy, safe, and innovative AI applications in healthcare.

1. Introduction

The healthcare sector is one of the most complex and demanding service provision systems globally. It is one of the most regulated sectors due to the very important good that it is called to serve and protect, which is that of human life and well-being in general.

Healthcare in the USA is heavily regulated at both federal and state levels, involving a complex mix of public and private entities and payors [1]. It is anomalous to refer to the US’s fragmented approach to healthcare as a system [2]. There is no constitutional right to healthcare. The federal government has numerous major programs that provide direct care or help finance healthcare services for citizens and some legal residents.

There is no ‘European Union health system’, but there is an ‘EU health policy’ [2]. The European Union affects the health of its citizens, the health of people around the world, and the operation and finance of its Member States’ healthcare systems in many ways. The EU health policy as such, with health as its declared objective, began in the 1980s. The 1986 Single European Act created the 1992 program of market integration. It involved a long list of harmonizing measures that would mean Member States, once they had hit an EU-wide regulatory minimum, would mutually recognize each other’s regulations. As part of these measures were some of the first European policies affecting healthcare, including the start of European regulation of pharmaceuticals and medical devices [3].

As artificial intelligence diffuses into everyday lives at a fast pace, these systems are offering new opportunities for healthcare service delivery, clinical care, and public health and promoting healthy individuals and patients. Integrating these AI-enabled health technologies, therefore, requires increased due diligence as they become integrated into various processes in the healthcare lifecycle concerning both healthy individuals and persons requiring medical assistance.

While AI-enabled health technologies are quickly evolving, this technology brings its own set of challenges that must be mitigated throughout their design, development, deployment, and maintenance lifecycles.

One considerable issue is how existing biases within these technologies may exacerbate health disparities. Income inequality, ‘technological discrimination’ [4] (e.g., age-related, geolocation-related, etc.), and ‘technological literacy’ can hamper the introduction and wider use of such technologies as well as increase population disparities and access to the latest advancements. There are significant technological limitations, including exacerbating care biases [5], privacy and security issues, and lack of transparency [6,7,8].

The current economic challenge for health systems is how to ensure affordable and equitable access to and use of the benefits offered by the AI tools, keeping in mind costly AI infrastructure, electricity demand, and water consumption [9,10].

Another prominent challenge is the lack of an appropriately skilled workforce in the field of health data science and in the field of design, development, and deployment of algorithmic systems for healthcare. According to the systematic review on ‘Frameworks for procurement, integration, monitoring, and evaluation of artificial intelligence tools in clinical settings’ by Khan et al. [11], the existing frameworks on AI implementation largely focus on the initial stage of implementation. Healthcare professionals repeatedly cite how challenging it is to implement AI in their clinical settings with little guidance on how to do so. The most common themes reported in the review were the rationale for use and legal liability for harm. Without a clear problem statement and rationale for use, the adoption of AI is unlikely. Unfortunately, existing frameworks do not yet emphasize the importance of deeply understanding and articulating the problem addressed by an AI tool.

Trust is a key factor influencing interactions between human beings, including their interactions with AI. Understanding the trust dynamics between AI and humans is crucial, particularly in the field of healthcare, where life is at risk. Many studies highlight the importance of understanding and addressing trust-related factors to ensure the effective integration of AI into healthcare practices [12,13].

This paper provides, among others, an indicative overview of the current regulatory landscape, discusses ethical and legal compliance issues in the European Union (EU) and the United States (USA), and proposes questionnaires to guide the decision-makers, the developers, and the implementers of AI systems.

2. Research Hypotheses

Given the exploratory and analytical nature of this study examining the development and implementation of safe, ethical, and trustworthy AI systems in healthcare, we developed the following research hypotheses:

Hypothesis H1:

Through comprehensive analysis of EU and US regulatory landscapes, international guidelines, and expert consultation, it is possible to develop structured questionnaires that operationalize regulatory and ethical requirements for both AI developers and implementers in healthcare settings.

Hypothesis H2:

The regulatory requirements and responsibilities for AI in healthcare differ substantially between developers (pre-market phase) and implementers (post-market phase), necessitating distinct assessment frameworks and guidance for each stakeholder group.

Hypothesis H3:

Current regulatory frameworks in the EU and US, while comprehensive in addressing technical compliance and safety requirements, contain gaps in areas such as liability allocation, AI-specific accreditation standards, and guidance for emerging AI technologies like generative AI and AI agents.

Hypothesis H4:

The integration of AI systems in healthcare requires consideration beyond technical performance, encompassing ethical dimensions (accountability, transparency, human oversight), legal aspects (product liability, medical liability), and organizational factors (accreditation, training, governance).

Hypothesis H5:

A lifecycle approach to AI system assessment—from problem definition through deployment and monitoring—can provide a comprehensive framework for ensuring trustworthy AI implementation in healthcare settings. Methodological Hypothesis.

Hypothesis H6:

A multidisciplinary approach combining literature review, regulatory analysis, and expert consultation can effectively identify practical requirements and translate complex regulatory frameworks into actionable guidance for healthcare stakeholders.

These hypotheses guided our investigation into the minimum essential requirements for developing and implementing AI systems that are not only technically robust but also ethically sound, legally compliant, and clinically beneficial.

3. AI Ethics Concepts in Healthcare

3.1. Types of AI

AI is a broad term that encompasses different systems based on their application, design, and purpose. Both developers and implementers should pay attention to different characteristics of embedded AI, centralized AI, and generative AI systems.

AI systems can generally be categorized as either embedded AI (on-device) or centralized AI (network-connected to devices). Embedded AI refers to intelligence integrated directly into hardware or specific devices, often performing focused tasks locally without relying on external servers. It operates within devices like smartphones, IoT devices, wearables, and appliances. Due to hardware constraints, embedded AI has limited computational power and performs specific, predefined tasks, such as facial recognition in cameras or predictive maintenance in industrial sensors. An example of embedded AI in healthcare is portable ECG monitors for on-the-spot diagnostics. Advantages include fast response times, offline capability, and energy efficiency, while challenges involve limited scalability and complexity compared to cloud-based solutions.

Centralized AI, often referred to as backend AI, powers the infrastructure and processes behind-the-scenes operations of software applications. It typically runs on cloud or server systems and supports broader, more complex tasks. Centralized AI boasts high computational capacity, scalability, and the ability to handle complex tasks. In healthcare, a centralized AI application could analyze medical images, like X-rays, MRIs, and CT scans, to assist radiologists in detecting abnormalities such as tumors, fractures, or infections.

Generative AI (GenAI) systems are designed to create new, original content (such as text, images, audio, video, and code) by learning patterns in existing data. These systems employ deep learning models, like transformers, Generative Adversarial Networks (GANs), or Variational Autoencoders (VAEs). GenAI emphasizes creativity, synthesis, and innovation in addition to task execution or decision-making. While they require large-scale training data and high computational resources, they can pose ethical concerns, reinforce biases, and demand significant resources. Generative AI is revolutionizing fields like drug discovery by designing novel molecules and optimizing existing compounds to target specific diseases [14].

Finally, an AI agent [15] is a system or program that perceives its environment through sensors or input data, processes information to reason, plan, or learn, and acts on the environment to achieve specific goals, often autonomously or semi-autonomously. These agents are designed to operate in dynamic environments, adapting to changes, learning from feedback, and making decisions based on their objectives. AI agents can encompass various categories, including embedded AI, centralized AI, and generative AI systems, depending on their function and application. In healthcare, an example of an AI agent is a virtual health assistant designed to interact with patients and assist with healthcare-related tasks, such as symptom checking, appointment scheduling, medication reminders, and even providing mental health support.

3.2. AI Ethics Challenges

The integration of artificial intelligence (AI) in healthcare has raised significant ethical concerns. As AI algorithms are increasingly used to diagnose diseases, develop treatment plans, and make clinical decisions, it is essential to ensure that these systems are fair, transparent, and unbiased. One of the major ethical issues in AI-powered healthcare is the potential for algorithmic bias, which can lead to unequal treatment of patients based on their demographic characteristics. Additionally, concerns about patient data privacy and security, as well as accountability and liability, must be addressed.

‘Accountability’ and ‘liability’ are crucial concepts in AI ethics in healthcare. ‘Accountability’ refers to the responsibility of individuals or organizations to explain and justify their actions, decisions, or outcomes, while ‘liability’ involves being legally responsible for any harm or damage caused by one’s actions or decisions. In the context of AI, ‘accountability’ involves being transparent about how AI systems are designed, trained, and deployed, as well as providing explanations for their decisions and actions. ‘Liability’, on the other hand, involves being legally responsible for any adverse consequences resulting from the use of AI systems, such as medical errors or patient harm. These concepts align with the OECD’s Principles on Artificial Intelligence, which stress the importance of maintaining appropriate oversight and ensuring accountability for AI projects [16].

A key aspect of AI in healthcare is safety, which refers to the design, development, and deployment of AI systems that prioritize patient safety and minimize the risk of harm. AI safety is closely correlated with AI ethics, as it ensures that patients’ autonomy is respected and that AI systems do not cause harm, either intentionally or unintentionally. The key aspects of AI safety include reliability, robustness, transparency, and explainability, all of which are essential for building trust in AI systems. This emphasis correlates with the WHO’s focus on patient safety, advocating the safe deployment of digital health solutions [17].

The concepts of ethical, safe, and responsible AI are closely interconnected in a healthcare context. ‘Ethical AI’ refers to the development and deployment of AI systems that align with medical ethics, principles, and values, while ‘safe AI’ focuses on minimizing the risk of harm to patients and ensuring that AI systems are reliable, robust, and secure. ‘Responsible AI’ involves considering the broader social and environmental implications of AI systems and ensuring that they are designed and used in a way that is respectful of patient values and principles, resonating with the OECD principles promoting AI that respects human rights and democratic values [16].

Ultimately, trustworthy AI requires a combination of all three concepts: ethical, safe, and responsible AI. By prioritizing these concepts, we can build AI systems that are reliable, fair, and beneficial and that earn the trust of humans. This can be achieved by developing robust AI ethics frameworks that prioritize patient well-being, safety, and autonomy, and by ensuring that AI systems are designed and deployed in a transparent, explainable, and fair manner. By doing so, we can harness the potential of AI to improve healthcare outcomes, while minimizing the risks and ensuring that AI systems are used in a way that is respectful of patient values and principles.

Papadopoulou et al. [18] propose a set of questions-principles as a basis for a future certification methodology for the AI systems’ developers themselves to foster a trustworthy AI system while helping the developers to conceive, design, deploy, test, and maintain the AI systems’ ethics efficacy and robustness, and to help protect the AI systems’ developers from any liability risks emanating from the relevant legislative and regulatory texts, allowing them at the same time to innovate and provide citizens with high-fidelity AI systems beneficial for their health and well-being. The authors comment that as a human being is partly characterized by their DNA and biomarkers, similarly, an algorithm is defined by its design and its data input. For the purposes of the analysis, an algorithm used within a health and care context was considered a ‘living’ entity defined initially by its core design and is subsequently affected by its ‘environment’. In both the core design phase and the ‘surrounding’ environments the algorithm is going to ‘operate’ in, a set of principles, translated eventually into norms, must encompass its ‘existence’ in order to guarantee optimized algorithmic ethics efficacy, effectiveness, and safety, avoiding bias and discrimination and ensuring the best outcome, while simultaneously augmenting the medical personnel’s decision-making capacity and increasing the accuracy of results (e.g., diagnosis, treatment, monitoring)—the ultimate goal and purpose of its use being the preservation or improvement of healthy individuals’ or patients’ conditions. Due to the particular features of healthcare contexts, an algorithmic system’s efficiency should be enhanced and measured against its ethics effectiveness and safety, similar to medicinal products’ effectiveness and safety, to ensure the highest degree of accuracy and reliability in real-world settings and to eliminate the possibility of bias, which can lead to unfair or inaccurate results.

In brief, the integration of AI in healthcare requires a multifaceted approach that prioritizes patient well-being, safety, and autonomy. To achieve this, AI systems must be designed with robust ethics frameworks that ensure fairness, transparency, and accountability. The concepts of accountability, liability, and safety are crucial in AI ethics, and their distinction and correlation are essential for trustworthy AI development. Connecting these principles to established international frameworks like those from the WHO and OECD underscores the need for global alignment to ensure AI technologies in healthcare are ethical, safe, and accountable [16,17].

3.3. Accountability of AI-Driven Healthcare Decisions

Accountability in AI-driven healthcare refers to the responsibility and transparency in the development, deployment, and use of artificial intelligence systems to ensure patient safety, ethical decision-making, public trust, and regulatory compliance [19]. To address accountability in AI-enabled healthcare, systems should establish clear responsibility frameworks that identify who is liable for validating their design, implementation, and continuous improvements. Given the interdisciplinary nature of these systems, these frameworks should also include roles and responsibilities for multiple stakeholders, including healthcare providers, AI developers, and regulatory bodies.

One fundamental approach is ensuring rigorous validation and monitoring of AI models both before and after deployment. Implementing an audit trail for AI decisions allows healthcare providers to trace each AI-driven decision, offering transparency and explainability and a basis for reviewing outcomes if errors occur. The audit across systems should detail the data inputs, processing stages, and reasoning behind AI outputs, providing a trail to understand and rectify potential errors. Collaborative oversight with multidisciplinary teams—including clinicians, ethicists, legal experts, and regulators—ensures that accountability is shared and balanced with clinical insights, regulatory compliance, and ethical standards. Another vital strategy is to incorporate ‘human-in-the-loop’ mechanisms, which ensure that clinicians retain the final responsibility over critical decisions, allowing them to validate or override AI recommendations as needed. AI systems, like human clinicians, are susceptible to error, underscoring the importance of shared accountability and robust oversight mechanisms. In this case, similar to shared accountability and audit-trail requirements for AI, there is a potential benefit for ‘human-in-the-loop’ to not only improve or protect from risks of AI errors, but also protect from human errors. In other words, using an auditable, transparent, and monitored system will not only improve or protect from AI errors but also help improve the accountability through evidence-based decision-making for the entire care ecosystem. Regulatory frameworks, such as the European Union’s Artificial Intelligence Act [20], emphasize assigning responsibility for adverse outcomes and mandating periodic audits to reinforce accountability. These strategies collectively create a layered accountability structure, reducing the risks associated with AI errors and ensuring that responsibility is clearly assigned across the AI lifecycle—from development to deployment and clinical use [6,21].

3.4. Balancing Accountability with Innovation

Accountability in applying any AI tool ultimately rests on the user of the tool. Furthermore, this accountability is nested in an ecosystem of regulatory agencies, professional organizations, software developers, and third-party implementers that can support and encourage innovation. This delicate balance between individual and organizational accountability that leverages industry innovation requires cooperation among all stakeholders within this ecosystem. As the former head of the U.S. Food and Drug Administration (FDA), Robert Califf acknowledged that the agency cannot and does not have the resources to monitor every possible problem that could arise within the FDA’s domain. As such, they rely on clear channels of communication and governance within this ecosystem to ensure the safety of any product. Without this fine-tuned balance, unintended harm can occur, such as when an algorithm cannot be generalized to an entire population or when the model that is the basis of a prediction tool has drifted in ways that had not been accounted for [22].

3.5. Balancing Innovation and Patient Safety

Innovative health technologies offer much to patients, clinicians, and health systems. In healthcare, innovation may be a novel idea, product, service, or care pathway that has clear benefits when compared to what is currently done. Successful innovations often possess two key qualities: they are both usable and desirable. Innovation can be defined as invention + adoption + diffusion [11]. Diffusion of innovation in the health and care sector is usually slow due to the complexity of the sector and demanding service provisions. It may take more time to embrace innovation for many reasons, such as a less robust body of evidence, perceived high costs, competing incentives, and a fear that once innovative technologies enter the health system, they will be difficult to remove [10,11]. Healthcare innovation is, however, not unique in slow adoption and diffusion. It took the telephone 64 years, electricity 45 years, computers 23 years, mobile phones 16 years, radio 12 years, and the Internet 13 years to achieve 40% consumer adoption [11,13].

Artificial intelligence is widely considered an innovation, as it is a technology that enables the creation of new products, services, and processes, significantly impacting various industries and driving positive change through its ability to analyze data, automate tasks, and enhance decision-making capabilities. According to the European AI Act, the ‘AI system’ is a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments [23].

3.6. Product Liability of AI Systems

In the European Union, the product liability of AI systems in healthcare is governed by the updated EU Product Liability Directive (Directive (EU) 2024/2853), which introduces stricter accountability for harm caused by defective AI products. Within this framework, developers of AI systems may be classified as ‘producers’, bearing strict liability if their technology causes personal injury, death, or property damage, regardless of fault. This includes liability for software defects, algorithmic bias, or lack of transparency in AI decision-making. Implementers, such as hospitals or healthcare providers, may also be held liable if the AI system is misused, improperly integrated, or modified outside its intended purpose, thereby introducing or exacerbating risks. To mitigate liability, both developers and implementers must ensure compliance with applicable EU regulations—including the AI Act, Medical Device Regulation (MDR), and General Data Protection Regulation (GDPR)—by conducting thorough risk assessments, maintaining transparency and documentation, and adhering to post-market monitoring obligations. The evolving liability landscape emphasizes the importance of a proactive, life-cycle approach to risk management and regulatory alignment for safe and trustworthy deployment of AI in healthcare.

In the United States, product liability law holds manufacturers, distributors, and retailers accountable for injuries or damages caused by defective products. Nevertheless, software is not considered a ’product’ for purposes of product liability law; rather, it is considered a ’service’, and thus outside the ambit of product liability law. That is important because product liability law is effectively a strict liability regime, whereas, otherwise, the usual negligence framework applies. As artificial intelligence becomes increasingly integrated into consumer products, questions arise about how existing product liability laws apply to AI technologies.

4. Materials and Methods

A comprehensive literature search was conducted across PubMed, Embase, Web of Science, and CINAHL Complete to identify relevant studies on AI in health systems. The search strategy included a combination of keywords, such as ‘artificial intelligence’, ‘health systems’, ‘health technology assessment,’ and ‘regulatory compliance’. The search was limited to studies published in English and focused on the period from 2010 to 2025.

The inclusion criteria for the studies were as follows:

• Studies that focused on the application of AI systems in health;
• Studies that focused on the development of AI systems in health;
• Studies that discussed the benefits and challenges of AI systems in health;
• Studies that provided a comprehensive review of regulatory compliance of AI systems in health.

The exclusion criteria for the studies were as follows:

• Studies that did not focus on AI systems;
• Studies that were not published in English;
• Studies that were not peer-reviewed;
• Studies that were published before 2010.

The data extraction process involved identifying relevant information from the included studies, including the study design, population, intervention, outcomes, and conclusions. The extracted data were then analyzed using a combination of qualitative and quantitative methods to identify patterns and themes related to AI systems in health.

The analysis involved the following:

• Thematic analysis to identify the benefits and challenges of AI systems in health;
• Content analysis to examine the regulatory compliance of AI systems;
• Network analysis to visualize the relationships between different concepts related to AI in health systems.

Overall, the literature search and data extraction process aimed to provide a comprehensive understanding of AI systems in health and their implications for regulatory compliance.

The methods applied were extensive literature research, the results from the discussions with AI developers, implementers, researchers, and policy-makers based on a short questionnaire (Box 1), and combining the co-authors’ multidisciplinary knowledge in medicine, machine learning, law, ethics, health systems, health informatics, and health innovation. The answers were summarized and used to draw conclusions. The anonymity and confidentiality were respected when reporting the findings.

Box 1. Issues covered

Aim:

• Fill information gaps and verify best practices mapped during a review of publicly available information
• Capture perceptions regarding challenges and trends

Questions:

What is AI (in healthcare)?

How is AI implemented?

• What are the technical implications and lessons learned?
• How is the ethics perspective taken into account (data privacy)?
• What are the current/envisaged AI governance frameworks?
• Is the implementation of AI in healthcare regulated and how (data privacy, liability, procurement) at State level/at Federal level?
• What kind of guidelines/checklists exist and how are they enforced?

Methodology for Interview Question Development

The semi-structured interview questions were developed using a systematic approach that combined the following:

• Literature Review Foundation: Initial questions were informed by our preliminary review of existing literature and competency frameworks, which revealed fragmented understanding of optimal AI technologies related to ethics, legal and regulatory knowledge, and competencies within healthcare education or medical practice settings.
• Research Team Expertise: Questions were collaboratively developed by the multidisciplinary research team, drawing on the following:
  • Clinical and medical education expertise (senior academics, medical educators, and practicing physicians)
  • Healthcare professional and research methodology experience (healthcare professional with extensive realist research experience)
  • Educational technology expertise (education adviser with technology focus)
  • Information technology professional perspective (senior IT professional)
• Stakeholder-Specific Adaptation: The core question framework was adapted for each stakeholder group (students, medical educators, digital sector experts) to ensure relevance while maintaining consistency across groups for comparative analysis.
• Exploratory Design Principles: Following our qualitative interpretivist and inductive research design, questions were deliberately broad and open-ended to encourage brainstorming and discussion, allowing themes to emerge naturally from participants rather than being constrained by predetermined categories.

Relevance and Rationale

These questions are relevant because they directly address our research objective: “How to Develop and Implement Safe, Ethical and Trustworthy AI Systems”. The exploratory style with prompting for brainstorming was essential for identifying minimum essential competencies from multiple perspectives without imposing existing frameworks that might not be contextually appropriate.

5. Results

This section presents our findings in relation to the research hypotheses, demonstrating how regulatory analysis can be translated into practical frameworks for AI in healthcare.

5.1. Validation of Research Hypotheses

Our analysis confirmed H1 by successfully developing two comprehensive questionnaires (Appendix A and Appendix B) that operationalize complex regulatory requirements. The developer questionnaire contains 56 questions across 12 domains, while the implementer questionnaire includes 18 questions across 6 domains, providing structured guidance for both stakeholder groups.

H2 was validated through our findings showing distinct regulatory obligations for developers versus implementers. As illustrated in Figure 1, developers bear responsibility across the entire AI lifecycle with extensive requirements summarized in Table 1, while implementers focus on deployment, monitoring, and clinical integration aspects detailed in Table 2.

The validity of H3 emerged through our analysis revealing regulatory gaps, particularly in AI-specific hospital accreditation (Section 6.2) and uncertainties in liability allocation between physicians and AI developers (Section 6.3). These findings highlight areas requiring further regulatory development.

H4 was confirmed through our comprehensive framework addressing multiple dimensions beyond technical requirements, including ethical considerations (Section 3), accountability mechanisms (Section 3.3), and the need for human oversight throughout the AI lifecycle.

Our lifecycle approach (H5) is reflected in Figure 1, which maps the continuous journey from problem definition through post-market surveillance, emphasizing that trustworthy AI requires ongoing assessment rather than one-time approval.

Finally, H6 was validated through our methodology combining extensive literature review, analysis of EU and US regulations, and expert consultation, resulting in practical questionnaires that bridge the gap between complex regulations and real-world implementation.

5.2. Regulatory Compliance of AI Systems in Healthcare

This paper examines the regulatory landscapes in the European Union (EU) and the United States (USA), drawing from the literature, international guidelines, professional associations, and expert recommendations. It proposes structured questionnaires for AI developers and implementers, designed to operationalize regulatory and ethical expectations and guide the development and deployment of trustworthy AI systems in healthcare. These questionnaires can also inform health technology assessments, public procurement, accreditation processes, and training programs for healthcare professionals. This section presents the results and discusses their implications for the healthcare sector (Appendix A: Questionnaire for AI Developers and Appendix B: Questionnaire for AI Implementers).

Regardless of the type of AI system, its implementation should be seen as a health system transformation lever, rather than a discrete set of technological devices [24]. It should encompass the whole lifecycle of an AI system as represented in Figure 1. This paper suggests that the assessment of AI value proposition should go beyond its technical/clinical performance, legal aspects, and cost logic. Ideally, similar to the philosophy of the ‘Health Technology Assessment (HTA) core model’ [25], a holistic analysis of its value in a real-world context of care and services should be performed. The analysis should include safety, ethical aspects, organizational aspects, and patient and social aspects. For better understanding, the paper describes separately the roles and responsibilities of developers and those of the implementers.

In 2024, the European Union enacted the world’s first comprehensive legal framework for AI, the EU AI Act (AIA), which covers all sectors and prioritizes safety and respect for fundamental human rights. In this context, the EU AI Act emerges as a pioneering legislative framework, marking the first comprehensive and general legal regulation of AI globally. Similar to the impact of the EU General Data Protection Regulation (GDPR) on any product or service involving personal data, the EU AI Act is poised to have a profound impact on the future of AI across industries [26]. Healthcare organizations faced substantial financial and administrative challenges in aligning with GDPR requirements [27,28]. However, the implementation of GDPR catalyzed greater patient trust, improved data accuracy, improved cybersecurity measures, protected patient information from unauthorized access, facilitated more consistent data management practices across healthcare institutions, and increased accountability for data handling, which led to improved governance and oversight.

The AIA, enacted in 2024, establishes a comprehensive framework to regulate AI systems, prioritizing safety and respect for fundamental rights. In the AIA, AI systems are classified into four categories based on risk—unacceptable, high, limited, and minimal—each with specific regulatory requirements. Medical devices are categorized as high-risk AI systems. The AIA emphasizes transparency, mandating that AI providers disclose relevant information, especially for high-risk applications, and sets out data governance requirements to minimize biases. It also includes a public database of high-risk AI systems and mandates oversight through compliance assessments. Additionally, the AI Act aims to ensure that AI systems are safe, fair, and aligned with EU values [12]. In his paper, Ebers [29] examines the European Union’s AI Act, emphasizing the need for a genuinely risk-based regulatory approach. The AI Act classifies AI systems into categories based on risk, but Ebers argues that the AIA lacks a true risk-benefit analysis and relies on predefined categories that may not accurately reflect real-world risks. He points out that current provisions could lead to over-regulation in low-risk applications and under-regulation in potentially harmful areas. Yet the guidelines and implementing acts will need to provide further guidance on how to improve AI governance under AIA. Despite the aim of the AI Act to address risks associated with predictive and generative AI (such as bias, discrimination, and misinformation), the Act relies heavily on self-regulation and includes numerous exemptions according to Wachter’s paper, “Limitations and Loopholes in the EU AI Act and AI Liability Directives”. Wachter highlights how the AIA and related liability directives often ignore non-material harms, like societal or emotional impacts, and lack robust mechanisms to address such issues. She also notes that the reliance on FLOPS (computational power) to define high-risk AI is inadequate, as it excludes many models with potentially harmful effects. The paper concludes by suggesting reforms, such as mandatory third-party audits, broader definitions of harm, and ethical transparency measures, to improve AI governance and mitigate the risks posed by these technologies across the EU and globally [30].

These recommendations are also echoed by Ebers, who also suggests that future-proofing the AI Act requires flexibility, such as guidelines and implementing acts that adapt to technological advancements. He also calls for empirical evidence in assessing AI risks and a framework to balance potential harms with societal benefits. His recommendations include empowering the European Commission to amend risk categories based on updated evidence, promoting harmonized standards, and ensuring proportional regulatory burdens. This approach, Ebers argues, would better balance innovation with fundamental rights and align with the EU’s principle of legislative proportionality. All of this assumes that risk categories are known and predictable. Complex systems are, however, unpredictable and must, therefore, have continuous outcome monitoring with feedback loops of correction in the algorithms or systems. Holland’s landmark paper on complex adaptive systems provides a theoretical foundation for designing AI models that are adaptive, self-correcting, and capable of handling the complexities of modern healthcare [31].

Regulatory Compliance Processes

Regulatory compliance of health technologies is a necessary requirement before they reach their intended audience. Legal regulation of AI will play a big part in the future of regulated digital medical products, including medical devices. The overarching legal framework surrounding AI/ML-enabled medical products is crucial, as it shapes their development, authorization, market introduction, deployment, and use. Until recently, medical devices were regulated primarily by sector-specific laws and regulations such as the FDA Medical Device Law and the EU Medical Devices Regulation (MDR). In the European Union, this has now changed with the EU AI Act [23] and the new Health Technology Assessment Regulation (HTAR). HTAR will streamline health technology assessments (HTA) across EU Member States, focusing on assessing the relative clinical effectiveness of health technologies like medical devices and treatments. It will standardize the HTA process by establishing joint clinical assessments at the EU level. While it supports innovation by easing market access, Member States retain autonomy in non-clinical assessments such as cost, ethical assessment, and reimbursement decisions [32,33]. Several authors addressed the importance of adapting the health technology assessment to digital health technologies, including AI [34,35,36].

5.3. Requirements for Developers

Developers, according to the CHAI (Coalition for Health AI) Assurance Standards Guide [37], include data scientists, machine learning engineers, and software developers that are responsible for creating and training AI models and ensuring alignment with healthcare needs. Product managers also fall under this category. They oversee the AI development process to ensure adherence to regulatory and ethical standards. Here is a summary of important legislation and guidance for developers from both the EU and USA (Table 1).

Table 1. A summary of important legislation and guidance for developers from both the EU and USA. Authors used GenAI to assist in formatting Table 1.

Legislation Name	What Does It Do?	How Is It Useful for Developers?
EU Legislation and Guidance
European AI Act (AIA) [23]	Classifies AI systems by risk and regulates high-risk applications to ensure safety, transparency, and fundamental rights.	Establishes clear compliance pathways, requiring risk assessments, data governance, transparency, and conformity checks.
Medical Device Regulation (MDR) [38]	Regulates medical devices in the EU with stricter requirements for clinical evaluation, safety, and traceability.	Requires rigorous evidence and post-market surveillance for medical software classified as devices; ensures patient safety.
In vitro diagnostic medical device (IVDR) Regulation [39]	Regulates diagnostics like lab tests and reagents with a strong emphasis on risk-based classification and performance.	Mandates clinical evidence, labeling, traceability, and post-market monitoring—important for developers of diagnostic tools.
EU Product Liability Directive [40]	Establishes strict liability for producers of defective products, including digital and AI-driven goods, ensuring compensation for harm caused without needing to prove negligence.	Developers must ensure proper deployment, monitoring, and documentation of AI tools to avoid liability and demonstrate due diligence in mitigating product risk.
EU Clinical Trials Regulation (Regulation (EU) No 536/2014) [41]	Harmonizes clinical trial approval and oversight processes across the EU.	Facilitates multi-country trials and ensures participant safety, transparency, and data integrity.
Health Technology Assessment Regulation (HTAR) [42]	Coordinates clinical effectiveness assessments of health technologies across EU countries.	Reduces duplication, speeds up access to markets, and supports evidence-based reimbursement decisions.
Commission Nationale de l’Informatique et des Libertés” (CNIL) ‘Self-assessment guide for artificial intelligence systems’ [43]	Offers practical steps for compliance with GDPR when using AI systems.	Helps identify risks and ethical issues early in AI design and development.
Reform of the EU Pharmaceutical Legislation [44]	Aims to ensure timely access to safe and affordable medicines, fostering innovation and addressing shortages within the EU pharmaceutical sector.	Encourages the development of innovative medical solutions and streamlines regulatory processes, facilitating faster time-to-market for new therapies.
EHDS—European Health Data Space Regulation [45]	Enables secure sharing and access to health data across EU Member States.	Supports interoperability, innovation, and cross-border health services while ensuring data privacy.
General Data Protection Regulation (GDPR) [46]	Standardizes data privacy laws across the EU, enhancing individuals’ data rights and mandating secure, lawful, and transparent data processing.	Developers must implement privacy-by-design, conduct DPIAs for high-risk systems, obtain consent, and ensure secure data handling.
US Legislation and Guidance
The Health Insurance Portability and Accountability Act (HIPAA) [47]	Protects the privacy and security of health information and sets standards for electronic health data.	Developers must ensure compliance with Privacy and Security Rules, especially in healthcare apps or platforms.
FDA Regulations Relating to Good Clinical Practice and Clinical Trials [48]	Ensures ethical, legal, and scientific standards for clinical trials involving human subjects.	Guides trial conduct, data integrity, informed consent, and compliance with ethical requirements.
FDA guidance for developers of AI enabled medical devices [49]	Provides a regulatory framework for AI/ML-based Software as a Medical Device (SaMD), covering design, updates, and oversight.	Enables continuous innovation via PCCP while ensuring safety using GMLP and real-world monitoring.
State level product liability laws [50]	Holds manufacturers liable for defective products under state-specific rules.	Developers of AI systems must assess risk for potential liability due to defects, especially for consumer or health-facing products.

Table 1. A summary of important legislation and guidance for developers from both the EU and USA. Authors used GenAI to assist in formatting Table 1.

Legislation Name	What Does It Do?	How Is It Useful for Developers?
EU Legislation and Guidance
European AI Act (AIA) [23]	Classifies AI systems by risk and regulates high-risk applications to ensure safety, transparency, and fundamental rights.	Establishes clear compliance pathways, requiring risk assessments, data governance, transparency, and conformity checks.
Medical Device Regulation (MDR) [38]	Regulates medical devices in the EU with stricter requirements for clinical evaluation, safety, and traceability.	Requires rigorous evidence and post-market surveillance for medical software classified as devices; ensures patient safety.
In vitro diagnostic medical device (IVDR) Regulation [39]	Regulates diagnostics like lab tests and reagents with a strong emphasis on risk-based classification and performance.	Mandates clinical evidence, labeling, traceability, and post-market monitoring—important for developers of diagnostic tools.
EU Product Liability Directive [40]	Establishes strict liability for producers of defective products, including digital and AI-driven goods, ensuring compensation for harm caused without needing to prove negligence.	Developers must ensure proper deployment, monitoring, and documentation of AI tools to avoid liability and demonstrate due diligence in mitigating product risk.
EU Clinical Trials Regulation (Regulation (EU) No 536/2014) [41]	Harmonizes clinical trial approval and oversight processes across the EU.	Facilitates multi-country trials and ensures participant safety, transparency, and data integrity.
Health Technology Assessment Regulation (HTAR) [42]	Coordinates clinical effectiveness assessments of health technologies across EU countries.	Reduces duplication, speeds up access to markets, and supports evidence-based reimbursement decisions.
Commission Nationale de l’Informatique et des Libertés” (CNIL) ‘Self-assessment guide for artificial intelligence systems’ [43]	Offers practical steps for compliance with GDPR when using AI systems.	Helps identify risks and ethical issues early in AI design and development.
Reform of the EU Pharmaceutical Legislation [44]	Aims to ensure timely access to safe and affordable medicines, fostering innovation and addressing shortages within the EU pharmaceutical sector.	Encourages the development of innovative medical solutions and streamlines regulatory processes, facilitating faster time-to-market for new therapies.
EHDS—European Health Data Space Regulation [45]	Enables secure sharing and access to health data across EU Member States.	Supports interoperability, innovation, and cross-border health services while ensuring data privacy.
General Data Protection Regulation (GDPR) [46]	Standardizes data privacy laws across the EU, enhancing individuals’ data rights and mandating secure, lawful, and transparent data processing.	Developers must implement privacy-by-design, conduct DPIAs for high-risk systems, obtain consent, and ensure secure data handling.
US Legislation and Guidance
The Health Insurance Portability and Accountability Act (HIPAA) [47]	Protects the privacy and security of health information and sets standards for electronic health data.	Developers must ensure compliance with Privacy and Security Rules, especially in healthcare apps or platforms.
FDA Regulations Relating to Good Clinical Practice and Clinical Trials [48]	Ensures ethical, legal, and scientific standards for clinical trials involving human subjects.	Guides trial conduct, data integrity, informed consent, and compliance with ethical requirements.
FDA guidance for developers of AI enabled medical devices [49]	Provides a regulatory framework for AI/ML-based Software as a Medical Device (SaMD), covering design, updates, and oversight.	Enables continuous innovation via PCCP while ensuring safety using GMLP and real-world monitoring.
State level product liability laws [50]	Holds manufacturers liable for defective products under state-specific rules.	Developers of AI systems must assess risk for potential liability due to defects, especially for consumer or health-facing products.

How are medical devices with AI placed on the market in the European Union? (Figure 2).

In the European Union, medical devices consisting of or using artificial intelligence systems must undergo a conformity assessment to demonstrate they meet legal requirements to ensure they are safe and perform as intended according to the Medical Devices Regulation [38] and the In vitro Devices Regulation [39] and the horizontal requirements of the EU AI Act Regulation [23]. They are regulated at the EU Member State level, but the European Medicines Agency (EMA) is involved in the regulatory process for certain types of medical devices. Manufacturers can place a CE (Conformité Européenne) mark on a medical device once it has passed a conformity assessment. The conformity assessment usually involves an external audit of the manufacturer’s quality system and, depending on the type of device, a review of technical documentation from the manufacturer on the safety and performance of the device. EU Member States designate accredited notified bodies [51] to conduct conformity assessments. For certain high-risk devices, notified bodies must request an opinion from specific expert panels before issuing a CE certificate. Demonstration of conformity is challenging for AI-based products, especially products that use sophisticated stochastic machine learning techniques, rather than for products whose software contains explicit, deterministic algorithms. Therefore, the European Association of Medical Devices Notified Bodies published a guideline/questionnaire [52] intended to provide AI-specific action guidance to industry in demonstrating conformity.

Healthcare AI systems, especially those impacting patient health and treatment, are subjected to detailed scrutiny. They must maintain high standards for accuracy, cybersecurity, robustness, and human oversight to minimize risks to users and patients. For example, systems used in patient triage or emergency response are classified as high-risk due to the critical nature of decisions they influence. These systems must also be registered in a public database, allowing for transparency and facilitating oversight by EU regulatory bodies. Additionally, the AI Act mandates that high-risk healthcare AI systems undergo a fundamental rights impact assessment, focusing on potential effects on users’ rights and ethical concerns and monitoring throughout the AI system’s lifecycle [23]. This requirement aims to proactively address issues related to bias and fairness, particularly in high-stakes medical applications.

How are medical devices with AI placed on the market in the United States? (Figure 2)

Before medical hardware or software is legally made available in the US market, the parent company must submit it to the United States’ Food and Drug Administration (FDA) for evaluation. For medically oriented AI/ML-based algorithms, the regulatory body has three levels of clearance, namely ‘pre-market clearance’ (510(k) [53]), ‘De Novo classification’ [54], or ‘pre-market approval’ [55]. The FDA’s traditional paradigm of medical device regulation was not designed for adaptive artificial intelligence and machine learning technologies. Many changes to artificial intelligence and machine learning-driven devices may need a pre-market review. On 6 January 2025, the FDA published the Draft Guidance for Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. This draft guidance proposes both lifecycle considerations and specific recommendations to support marketing submissions for AI-enabled medical devices. Key novelties include a ‘Comprehensive Total Product Life Cycle (TPLC) Approach’ ensuring that AI-DSFs are safe, effective, and of high quality throughout their entire lifecycle. This approach encompasses design, development, deployment, and maintenance phases, promoting continuous oversight and improvement. It provides manufacturers with specific guidance on the content of marketing submissions. This includes comprehensive documentation on device descriptions, risk management plans, model details, and data management strategies, ensuring thorough evaluation of safety and effectiveness. The guidance introduces the concept of a ‘Predetermined Change Control Plan (PCCP)’, allowing manufacturers to implement planned modifications to AI models without necessitating repeated FDA authorizations. This facilitates timely updates while maintaining regulatory compliance. The U.S. Food and Drug Administration also places focus on transparency and bias mitigation. Manufacturers are encouraged to design transparent AI models and utilize representative datasets to mitigate biases, thereby promoting equity in healthcare delivery. The draft guidance also underscores the importance of continuous performance monitoring to detect issues such as data drift or decreased model reliability. Additionally, it highlights the need for robust cybersecurity measures to protect devices and data throughout their lifecycle [56]. However, in the recent JAMA paper, the former FDA Commissioner Gottlieb expresses concern over new FDA policies hindering the capabilities of AI tools with advanced analytical capabilities used in clinical practice [57].

5.4. Requirements for Implementers

Implementers and end users, according to CHAI (Coalition for Health AI) Assurance Standards Guide [58], include healthcare providers and administrators, clinicians, and healthcare professionals. They are responsible for the integration of AI solutions into clinical workflows and for managing the operational adoption. Clinicians and healthcare professionals use AI tools for diagnosis, treatment planning, and workflow optimization. Hospitals and healthcare organizations would need to ensure that AI-driven decisions adhere to state-of-the-art medical guidelines, assessing AI bias, explainability, and transparency; ensuring AI systems comply with HIPAA (data privacy), FDA regulations (if AI is a medical device), and CMS (Medicare/Medicaid); ensuring staff understand how AI decisions are made and can override them if necessary; and ensuring they have robust error reporting, bias monitoring, and continuous improvement in place. Here is a summary of important legislation and guidance for implementers from both the EU and the USA (Table 2).

Table 2. A summary of important legislation and guidance for implementers from both the EU and USA. The authors used GenAI to assist in formatting Table 2.

Legislation Name	What Does It Do?	How Is It Useful for Implementers?
EU Legislation and Guidance
General Data Protection regulation (GDPR) [46]	Standardizes data privacy laws across the EU, enhancing individuals’ data rights and mandating secure, lawful, and transparent data processing.	Implementers must ensure systems securely handle personal data, obtain valid consent, and meet data subject rights obligations.
European AI Act (AIA) [23]	Classifies AI systems by risk and regulates high-risk applications to ensure safety, transparency, and fundamental rights.	Implementers must monitor deployment for conformity, ensure transparency, and maintain human oversight in high-risk applications.
Health Technology Assessment Regulation (HTAR) [42]	Coordinates clinical effectiveness assessments of health technologies across EU countries.	Implementers must align deployments with evidence expectations for reimbursement and decision-making across Member States.
EU Clinical Trials Regulation (Regulation (EU) No 536/2014) [41]	Harmonizes clinical trial approval and oversight processes across the EU.	Implementers must ensure that AI tools used in trials meet centralized authorization, safety reporting, and transparency requirements.
EU Product Liability Directive [40]	Establishes strict liability for producers of defective products, including digital and AI-driven goods, ensuring compensation for harm caused without needing to prove negligence.	Implementers can still bear responsibility if they modify an AI system beyond its intended use, if their integration or deployment of the system introduces a defect, and if they fail to monitor or properly train users, leading to misuse.
EHDS—European Health Data Space Regulation [45]	Enables secure sharing and access to health data across EU Member States.	Implementers must ensure systems are compatible with cross-border data sharing standards, support secure data access for healthcare delivery and research, and comply with governance rules for ethical secondary data use.
US Legislation and Guidance
The Health Insurance Portability and Accountability Act (HIPAA) [47]	Protects the privacy and security of health information and sets standards for electronic health data.	Implementers must ensure systems enforce access controls, audit logs, encryption, and incident response mechanisms.
FDA Regulations Relating to Good Clinical Practice and Clinical Trials [48]	Ensures ethical, legal, and scientific standards for clinical trials involving human subjects.	Implementers must use validated AI systems in trials and follow oversight, informed consent, and reporting rules.
21st Century Cures Act (2016) [58]	Modernizes healthcare regulation and exempts some clinical decision support software from device regulations.	Implementers must determine the regulatory status of AI tools and ensure proper transparency for non-regulated CDS tools.
AI Bill of Rights (2022) [59]	Outlines principles for responsible AI use, including safety, privacy, fairness, and transparency.	Implementers must evaluate deployed AI for potential harms and bias and ensure informed user engagement.
Algorithmic Accountability Act [60]	Proposes mandatory risk assessments and impact audits for automated systems.	Implementers may be required to assess system risks, document mitigation strategies, and report on algorithmic impact.
Equal Credit Opportunity Act (ECOA) and Fair Housing Act [61]	Prohibit algorithmic bias in credit and housing contexts, affecting healthcare AI related to access and coverage.	Implementers must avoid biased AI outputs affecting vulnerable groups and comply with anti-discrimination laws.
State-Level AI and Data Privacy Laws (e.g., California Consumer Privacy Act—CCPA) [62]	Regulate personal data use, AI profiling, and user rights at the state level.	Implementers must ensure AI system compliance with consent, explainability, and opt-out requirements.
NIST AI Risk Management Framework (2023) [63]	Provides voluntary best practices for trustworthy AI development and deployment.	Implementers can use it to assess and reduce system risks while promoting safe and ethical operations.
Federal Trade Commission (FTC) Guidelines on AI and Healthcare [64]	Advises against deceptive AI claims and biased systems in consumer health.	Implementers must ensure systems are transparent, non-misleading, and aligned with consumer protection laws.

Table 2. A summary of important legislation and guidance for implementers from both the EU and USA. The authors used GenAI to assist in formatting Table 2.

Legislation Name	What Does It Do?	How Is It Useful for Implementers?
EU Legislation and Guidance
General Data Protection regulation (GDPR) [46]	Standardizes data privacy laws across the EU, enhancing individuals’ data rights and mandating secure, lawful, and transparent data processing.	Implementers must ensure systems securely handle personal data, obtain valid consent, and meet data subject rights obligations.
European AI Act (AIA) [23]	Classifies AI systems by risk and regulates high-risk applications to ensure safety, transparency, and fundamental rights.	Implementers must monitor deployment for conformity, ensure transparency, and maintain human oversight in high-risk applications.
Health Technology Assessment Regulation (HTAR) [42]	Coordinates clinical effectiveness assessments of health technologies across EU countries.	Implementers must align deployments with evidence expectations for reimbursement and decision-making across Member States.
EU Clinical Trials Regulation (Regulation (EU) No 536/2014) [41]	Harmonizes clinical trial approval and oversight processes across the EU.	Implementers must ensure that AI tools used in trials meet centralized authorization, safety reporting, and transparency requirements.
EU Product Liability Directive [40]	Establishes strict liability for producers of defective products, including digital and AI-driven goods, ensuring compensation for harm caused without needing to prove negligence.	Implementers can still bear responsibility if they modify an AI system beyond its intended use, if their integration or deployment of the system introduces a defect, and if they fail to monitor or properly train users, leading to misuse.
EHDS—European Health Data Space Regulation [45]	Enables secure sharing and access to health data across EU Member States.	Implementers must ensure systems are compatible with cross-border data sharing standards, support secure data access for healthcare delivery and research, and comply with governance rules for ethical secondary data use.
US Legislation and Guidance
The Health Insurance Portability and Accountability Act (HIPAA) [47]	Protects the privacy and security of health information and sets standards for electronic health data.	Implementers must ensure systems enforce access controls, audit logs, encryption, and incident response mechanisms.
FDA Regulations Relating to Good Clinical Practice and Clinical Trials [48]	Ensures ethical, legal, and scientific standards for clinical trials involving human subjects.	Implementers must use validated AI systems in trials and follow oversight, informed consent, and reporting rules.
21st Century Cures Act (2016) [58]	Modernizes healthcare regulation and exempts some clinical decision support software from device regulations.	Implementers must determine the regulatory status of AI tools and ensure proper transparency for non-regulated CDS tools.
AI Bill of Rights (2022) [59]	Outlines principles for responsible AI use, including safety, privacy, fairness, and transparency.	Implementers must evaluate deployed AI for potential harms and bias and ensure informed user engagement.
Algorithmic Accountability Act [60]	Proposes mandatory risk assessments and impact audits for automated systems.	Implementers may be required to assess system risks, document mitigation strategies, and report on algorithmic impact.
Equal Credit Opportunity Act (ECOA) and Fair Housing Act [61]	Prohibit algorithmic bias in credit and housing contexts, affecting healthcare AI related to access and coverage.	Implementers must avoid biased AI outputs affecting vulnerable groups and comply with anti-discrimination laws.
State-Level AI and Data Privacy Laws (e.g., California Consumer Privacy Act—CCPA) [62]	Regulate personal data use, AI profiling, and user rights at the state level.	Implementers must ensure AI system compliance with consent, explainability, and opt-out requirements.
NIST AI Risk Management Framework (2023) [63]	Provides voluntary best practices for trustworthy AI development and deployment.	Implementers can use it to assess and reduce system risks while promoting safe and ethical operations.
Federal Trade Commission (FTC) Guidelines on AI and Healthcare [64]	Advises against deceptive AI claims and biased systems in consumer health.	Implementers must ensure systems are transparent, non-misleading, and aligned with consumer protection laws.

6. Discussion

As illustrated throughout the paper, introducing a new technology into the healthcare system follows certain steps. Regulatory compliance of health technologies is a necessary requirement before they reach their intended audience in the EU and in the USA. Then, in the European Union, a health technology will follow the new Health Technology Assessment Regulation and undergo a health technology benefit-harm assessment, where the following example questions will be answered: ‘How well does a new technology work compared with existing alternative health technologies? For which patients does it work best?’ It should undergo an economic evaluation to answer the following question: ‘What costs are entailed for the health system?’ Finally, legal frameworks for assigning responsibility and liability, and for potential ethical dilemmas, such as equitable access, should be addressed as well.

Several key pieces of legislation and guidance govern the implementation of AI solutions in US healthcare, focusing on data privacy, patient safety, clinical validation, bias mitigation, and liability. While HIPAA governs data security, the FDA regulates medical AI tools, and the Cures Act creates exemptions for decision-support software. Emerging regulations (like the AI Bill of Rights and Algorithmic Accountability Act) will further shape the landscape. Organizations implementing AI solutions must continuously monitor regulatory updates to ensure compliance and mitigate liability risks.

6.1. Product and Medical Liability and the Use of AI Systems

Medical liability in the context of AI systems intersects increasingly with tort law, particularly as healthcare providers integrate AI into clinical decision-making. Under tort law—especially the principles of negligence and product liability—both healthcare professionals and AI developers may be held accountable if the use of an AI system leads to patient harm. For example, if an AI diagnostic tool produces an incorrect recommendation that a clinician follows without adequate oversight, liability may arise due to a failure in exercising reasonable medical judgment. At the same time, the developer of the AI system may face strict liability if the software itself was inherently flawed or inadequately tested. As AI systems evolve from assistive tools to more autonomous roles, establishing standards of care, clarifying duty of care, and ensuring explainability and traceability become essential for mitigating liability. Ultimately, tort law provides the legal framework through which injured patients can seek compensation, emphasizing the need for transparency, compliance, and human oversight in AI-supported healthcare.

When does physician use of AI increase liability? paper discusses the relationship between a physician’s liability and the use of AI systems, in which circumstances physicians using AI systems are more likely to be found liable, and how physicians can reduce their potential liability. The results of the study showed that physicians who receive advice from an AI system to provide standard care could reduce the risk of liability by accepting (rather than rejecting) that advice. However, when an AI system recommends non-standard care, there is no similar shielding effect of rejecting that advice. The study also found that lay jurors rely on both factors, i.e., whether the treatment provided was standard and whether the physician followed the AI recommendation, in evaluating physicians who use medical AI systems. The conclusions of the study were that tort law may not impose as great a barrier to the uptake of AI medical system recommendations as is commonly assumed, and in fact, it might even encourage the uptake of AI recommendations [65].

The systematic review on ‘Defining medical liability when artificial intelligence is applied on diagnostic algorithms’ discusses the application of artificial intelligence (AI) in medicine, specifically in diagnostic algorithms, and the related issue of medical liability. The authors conducted a systematic review of the literature to identify the advantages and critical issues of AI in medicine, as well as the liability concerns. The paper also explored the challenges of apportioning liability in cases where AI-related errors occur, including the difficulty of identifying the responsible party and the need for clear regulations. The analysis demonstrated that the use of AI in medicine raises significant liability concerns and that the current regulatory framework is inadequate to address these issues. The authors suggested that a new conception of the doctor-patient relationship is needed, one that takes into account the role of AI as a ’diagnostic assistant’. They also proposed that hospitals and healthcare institutions may need to assume liability for AI-related errors and that specific insurance contracts may be necessary to cover this risk. The paper highlighted the need for further research and discussion on the medico-legal implications of AI in medicine and for the development of clear guidelines and regulations to address these concerns [66].

Shumway and Hartman, in their recent paper, discuss the potential liability concerns for physicians using the generative large language models (LLM) in healthcare. The authors reviewed past case law in the US related to third-party medical guidance and current regulations regarding medical malpractice liability in AI, highlighting the uncertainty and lack of clear precedent in this area. The main discussion points included the potential sources of liability for physicians using LLM AI, the role of the US Food and Drug Administration in regulating clinical AI reliability, and the need for proactive policy action to address these issues. The main conclusions of the analysis were that the lack of clarity on malpractice liability for LLM AI-influenced medical decisions poses a significant barrier to the adoption of this technology in medicine and that federal policy should mandate rigorous validation and testing of AI tools before their deployment in clinical settings. The authors also conclude that liability reform may be needed to shift some responsibility for AI-generated medical guidance to algorithm developers. The main recommendations were that the US FDA should extend its responsibility to regulate clinical AI reliability, including requiring AI developers to make their algorithms available for independent validation and ensuring that AI systems provide clear explanations for their recommendations with verified, peer-reviewed data. The authors also recommend that federal policy should encourage tort reform to share liability between physicians and LLM developers and that proactive action should be taken to address these issues through policy rather than waiting for resolution through the legal system [67].

With reference to the relationship between tort liability regimes and the liability emanating from AI uses in healthcare, Sullivan and Schweikart indicated that the application of traditional tort liability provisions applicable in the case to AI technologies is difficult because, as per another author, these provisions are designed and focused on human conduct, and thus do not function easily when a technology is used. The challenges also arise from the opaque nature of ‘black-box AI’, which makes it difficult to determine the reasoning behind its conclusions, as well as the difficulty of attributing and distributing legal responsibility. The authors concluded that current tort liability doctrines are insufficient to address the realities of AI-related medical malpractice and that new legal solutions are necessary to craft novel legal standards and models that address the nature of AI. They recommended conferring ’personhood’ on AI systems, allowing them to be sued directly for negligence claims with appropriate insurance coverage for their errors. They also suggest the implementation of a common enterprise liability theory, where all parties involved in the use and implementation of AI share responsibility for any injuries caused. The authors also proposed the modification of the standard of care for healthcare professionals using black-box AI, requiring them to exercise due care in evaluating and implementing AI algorithms [68].

The main findings of the O’Sullivan’s paper on the potential of AI and autonomous robotic surgery and the need for frameworks to ensure their safe and responsible development and use suggest that current legal frameworks may not be sufficient to address the complexities of autonomous robotic surgery and that new approaches to liability and culpability may be needed. Furthermore, the paper highlighted the importance of explainability and transparency in AI decision-making, particularly in high-stakes applications such as surgery. The main recommendations of the paper included the development of standards and regulations for autonomous surgical systems, the establishment of clear guidelines for accountability, liability, and culpability, and the promotion of explainability and transparency in AI decision-making [69].

A converging point among these papers is the emphasis on the importance of medical liability and the need for clear policies and regulations to protect healthcare professionals and patients. Another common point is the discussion on the need for transparency and accountability in medical decision-making, particularly when artificial intelligence is involved. The paper suggested that healthcare professionals and developers of AI systems must be aware of the potential risks and benefits of AI and take steps to ensure that patients are fully informed and protected.

Based on the above, a crucial question that could steer the discussions on liability when AI technologies are used by physicians or in healthcare contexts is how tort liability provisions can ensure an equitable distribution between a physician’s duty of care and a patient’s right to be informed and receive care that is augmented by powerful technology.

6.2. Accrediting AI Technologies for Hospitals

AI integration in accreditation is more advanced in the US, while Europe lags behind in regulatory adaptation. None of the hospital accreditation agencies specifically accredit AI technologies [69,70], but they assess hospital-wide policies, patient safety, and risk management, which include AI-driven systems.

Hospitals in the US are accredited through voluntary programs that evaluate their compliance with specific standards of healthcare quality and patient safety. Accreditation is voluntary but vital for quality assurance, patient trust, and funding eligibility. Multiple agencies offer accreditation [71,72,73,74]. Hospitals are evaluated on patient safety, staff qualifications, facility conditions, and performance. Hospitals undergo rigorous surveys, typically every 3 years, conducted by these organizations to ensure they meet federal and state regulations and quality standards. The accreditation process ensures that the hospitals adhere to strict protocols, reducing errors and improving patient outcomes. Without accreditation, hospitals may lose eligibility for government funding. Patients and insurers trust accredited hospitals for their adherence to high standards. Finally, hospitals must continuously improve policies, procedures, and safety measures to maintain accreditation. Nevertheless, none of the hospital accreditation agencies specifically accredit AI technologies [70,71], but they assess hospital-wide policies, patient safety, and risk management, which include AI-driven systems.

The role of hospital accreditation organizations in AI accreditation would include ensuring AI-driven decisions adhere to evidence-based medicine; assessing AI bias, explainability, and transparency; ensuring AI systems comply with HIPAA (data privacy), FDA regulations (if AI is a medical device), and CMS (Medicare/Medicaid); ensuring staff understand how AI decisions are made and can override them if necessary; and ensuring hospitals using AI have robust error reporting, bias monitoring, and continuous improvement in line with accreditation standards.

Historically, hospital accreditation in the EU aimed for voluntary, professionally driven continuing improvement, but since the mid-1990s, new and existing programs have increasingly become mechanisms for accountability to the public and to regulatory and funding agencies. There has been an increase in the number of countries engaging in hospital accreditation programs. Hospital accreditation aims to improve healthcare quality. Accreditation typically assesses process indicators like infection control, patient satisfaction, and diagnostic accuracy, but it does not consistently measure clinical outcomes. Many European countries have accreditation programs, with mostly national but also regional models. Programs vary between mandatory and voluntary, with increasing government involvement. Accreditation governance includes clinicians, hospital owners, and regulators, while evaluation methods combine self-assessments and scheduled external reviews, with unannounced inspections being rare [75,76]. Adherence to ISQua EEA standards is an emerging trend [77].

6.3. Requirements for AI Agents in Healthcare

While embedded AI, centralized AI, and generative AI focus on the technology or purpose, AI agents emphasize the role and behavior of the system within a broader environment. An AI agent could leverage any or all of these technologies, making it a versatile and application-oriented concept. An example of an AI agent in health is a virtual health assistant designed to interact with patients and assist with healthcare-related tasks, such as symptom checking, appointment scheduling, medication reminders, and even mental health support [78].

Since AI agents are designed to take on different roles and tasks that will affect the organization of healthcare delivery and services, the authors believe that the performance of AI agents should be rigorously evaluated and continuously monitored, as indicated in a recent paper by Rajpurkar and Topol [79]. For example, practicing physicians have to complete undergraduate education at an accredited medical school, undertake a residency program, obtain a medical license, take the specialty certification exam, and obtain board certification. Board certification is not a one-time event but a commitment to continuous professional development. The continuing certification programs promote lifelong learning, self-assessment, and practice improvement. These programs ensure that physicians maintain their knowledge and skills throughout their careers, adapting to advancements in medical practice and patient care. By adhering to this comprehensive certification process, physicians demonstrate their dedication to providing high-quality care and their commitment to ongoing professional development [80,81,82,83].

The authors recommend that special attention be paid to human oversight for semi-autonomous and autonomous AI systems to ensure, same as for health professionals, that they meet rigorous standards in their respective roles.

6.4. Ensuring Trustworthiness in Complex “Black Box” AI Models

The integration of artificial intelligence (AI) in healthcare, particularly complex “black box” models such as generative AI, poses significant challenges in ensuring trustworthiness. While transparency and explainability are crucial, simply mandating disclosure may not be sufficient to address the intricacies of these models. Generative AI, with its ability to create new, original content, introduces additional layers of complexity, making it challenging to understand how decisions are made.

One of the primary concerns with “black box” models is their opacity, which can lead to a lack of trust among healthcare professionals and patients. To mitigate this, techniques such as model interpretability, explainability, and transparency can be employed. Model interpretability involves making the model’s decisions and predictions understandable, while explainability focuses on providing insights into how the model works [84]. Transparency, on the other hand, requires that the model’s underlying data, algorithms, and decision-making processes be openly accessible.

However, implementing these techniques in complex AI models can be daunting. For instance, generative AI models, which rely on deep learning architectures, can be particularly challenging to interpret due to their non-linear and high-dimensional nature. Moreover, the lack of standardization in explainability and interpretability methods can hinder their widespread adoption.

To address these challenges, researchers and developers can explore innovative solutions, such as the following:

• Developing explainable AI frameworks: Creating frameworks that provide insights into the decision-making processes of complex AI models can help build trust among stakeholders.
• Implementing model-agnostic interpretability methods: Techniques that can be applied to various AI models, regardless of their architecture, can facilitate the interpretation of complex models.
• Establishing standards for transparency and explainability: Developing standardized guidelines for transparency and explainability can ensure consistency across different AI models and applications.
• Fostering collaboration between AI developers and healthcare professionals: Encouraging collaboration between AI developers and healthcare professionals can help ensure that AI models are designed with transparency, explainability, and trustworthiness in mind.

These directions support the practical refinement of AI frameworks, promoting accountable and ethically responsible AI systems in healthcare.

Finally, building a trustworthy environment for AI also depends on the organizational culture in which these technologies are implemented. Research suggests that supportive, transparent, and value-aligned workplaces foster greater employee authenticity, which enhances engagement and openness to innovation [85]. In healthcare, such a culture can increase trust in AI tools, promote ethical adoption, and ensure that professionals feel empowered to integrate these technologies into their practice with confidence. As such, investing in organizational culture should be seen as a foundational step in the responsible deployment of complex AI systems.

6.5. Case Study—Implementing AI Locally: Best Practice Example of University of Washington (UW) Medicine

This case study illustrates how the theoretical frameworks and regulatory requirements discussed throughout this paper translate into practical implementation at the institutional level.

While our analysis has focused on regulatory compliance requirements at the EU and US federal levels, the actual deployment of AI systems occurs within individual healthcare organizations. The UW Medicine example demonstrates how a major academic medical center operationalizes the complex regulatory landscape through structured governance processes, interdisciplinary oversight, and practical guidelines. This real-world implementation validates our proposed questionnaires and highlights how healthcare institutions can bridge the gap between regulatory requirements and clinical practice.

By examining UW Medicine’s approach to AI governance—including their GenAI Task Force, review processes, and interim guidelines—readers can understand how the abstract regulatory principles discussed in previous sections manifest in concrete organizational policies and procedures. This case study thus serves as a practical blueprint for other healthcare institutions seeking to implement AI technologies while maintaining compliance with regulatory standards and ethical principles.

In the United States as well as in the European Union, the piloting and implementing of AI systems in healthcare is handled at the local level of service provision.

Healthcare organizations in the USA are responsible for the integration of AI health technologies into the processes and workflows. For example, the University of Washington School of Medicine (UW Medicine) employs a structured and collaborative approach to pilot, procure, and implement AI-based systems in healthcare, ensuring alignment with institutional policies and regulatory standards. The process involves several key components.

UW Medicine has established interim guidelines in 2023 outlining the responsible use of generative AI and large language models (LLMs) within healthcare settings. These guidelines, developed by an interdisciplinary workgroup, specify acceptable use cases, data handling protocols, and compliance requirements, ensuring that AI tools are integrated ethically and securely. All generative AI pilot projects undergo a comprehensive review by the UW Medicine Generative AI (GenAI) Task Force [86]. This process includes proposal submission, preliminary review, and necessary evaluations by human resources/labor, legal, compliance, and information security teams. This structured oversight ensures comprehensive governance, from initial proposal through ongoing monitoring, allowing UW Medicine to responsibly integrate AI solutions that align with institutional standards, strategic objectives, and regulatory requirements. This framework supports high-quality patient care, reinforces safety, and enhances operational performance. This case study demonstrates a real-world example of successful AI integration, illustrating potential pathways for other healthcare institutions aiming to implement AI technologies responsibly and effectively.

7. Concluding Remarks

Many discussions revolve around the responsibility and liability of AI systems in healthcare settings and whether AI acts solely as a tool to facilitate innovation. While AI systems function as tools, their ability to learn, adapt, and shape decision-making makes them fundamentally different from traditional technologies. They act as dynamic assistants, decision influencers, and socio-economic forces rather than static instruments. AI is therefore an evolving system that interacts with human decision-making, governance structures, and societal frameworks.

Various steps will need to be taken to facilitate the evaluation and monitoring of AI systems and their integration into health systems. Existing regulatory processes will need to be adapted to address the disruptive and evolving nature of AI systems. Standard practices, where the machine learning community openly validates an AI model, should be encouraged and recognized [87].

Continuous monitoring and evaluation of AI systems’ performance are essential to detect and address issues such as data drift or decreased model reliability, as emphasized by current industry guidelines [87].

In the case of AI agents, adequate steps mimicking the physicians’ certification [79] and peer review should be undertaken to evaluate the knowledge and skills and provide patients with a trusted ‘human-centric’ healthcare environment. For instance, only AI agents that are certified, have gone through a ‘residency’ program, and have an active license that can be challenged based on adverse outcomes could be adopted into the system.

Developers need clarity and guidance on how to navigate the regulatory frameworks. Implementers lack evidence and need guidance on how to implement the new tools. AI is advancing so much faster than research and evidence. Researchers believe that more emphasis should be placed on gathering evidence through implementation science research and randomized clinical trials (RCT) of AI. Health systems experts call for health economic studies of AI. Health professionals should be included in the problem statement and development of AI systems. Health professionals also need tailored training on AI tools and on how to explain AI tools used in the domain of patient care to patients. Finally, patients lack adequate AI literacy and ask for transparency about when, what, and how AI is used in the healthcare process. This paper draws inspiration from the reported guidelines and recommendations of international organizations, professional associations [33,37,42,52,76,88,89,90,91,92], and experts and proposes questions for developers and for implementers (Annex 1) of AI systems. These questions can be translated in consultation with stakeholders into quantifiable checklists that can serve as guidance when developing AI systems and when implementing AI systems in hospitals and healthcare settings.

8. Potential Future Research Directions

Based on our findings and the limitations of this study, we propose several key research directions that address gaps identified in our analysis.

Our research revealed that while comprehensive regulatory frameworks exist, significant challenges remain in their practical implementation and evaluation. The following research directions emerge directly from our results and aim to address the study’s limitations:

Addressing Methodological Limitations:

Our study relied primarily on literature review, regulatory analysis, and limited expert consultation. To strengthen and validate our findings, empirical studies are essential to test the practical application of our proposed questionnaires and offer real-world insights into AI systems’ effectiveness and impact.

Research Directions Based on Key Findings:

• Pilot Studies for Questionnaire Validation: Our results produced comprehensive questionnaires (Appendix A and Appendix B), but these instruments require empirical validation. Future studies should test these questionnaires with actual AI developers and implementers to assess their practicality, completeness, and effectiveness. Mixed-method approaches combining quantitative usability metrics with qualitative feedback through interviews will be crucial to refine these tools and ensure they adequately capture all relevant regulatory and ethical considerations.
• Addressing Regulatory Gaps Identified: Our analysis revealed significant gaps in AI-specific hospital accreditation standards (Section 6.2) and uncertainties in liability allocation (Section 6.1). Future research should develop and test frameworks for AI-specific accreditation criteria and conduct comparative legal analyses across jurisdictions to propose clearer liability models for AI-related medical errors.
• Longitudinal Analysis of AI System Performance: While our study provides a snapshot of current regulatory requirements, it cannot assess long-term compliance or system evolution. Future research should track AI systems throughout their lifecycle, from initial deployment through multiple update cycles, to understand how regulatory compliance evolves and identify patterns of model drift or performance degradation that current frameworks may not adequately address.
• Comparative Effectiveness of Different Governance Models: Our case study (Section 6.5) presented one institutional approach, but broader comparative research is needed. Future studies should examine multiple healthcare institutions’ AI governance models to identify best practices and develop evidence-based recommendations for organizational structures that effectively balance innovation with safety.
• Quantitative Assessment of Regulatory Impact: Our discussion of GDPR’s impact (Section 4) was largely qualitative. Future research should quantitatively measure how comprehensive AI regulations affect healthcare organizations through metrics such as implementation costs, time to deployment, patient trust scores, and clinical outcomes. This empirical data will be crucial for policymakers to refine regulatory frameworks.
• Development of AI Agent Certification Frameworks: Our results highlighted the unique challenges posed by AI agents (Section 6.3) but could not propose detailed certification processes. Future research should develop and pilot certification frameworks for AI agents that mirror medical professional certification, including competency assessments, continuous monitoring, and recertification requirements.
• Longitudinal Analysis of AI Model Drift: This investigation can focus on AI model drift in healthcare applications, tracking performance indicators over time. Continuous monitoring will evaluate re-validation protocols’ effectiveness, informing best practices for AI system maintenance and risk mitigation.

Addressing Geographic and Contextual Limitations:

This study focused on EU and US regulatory frameworks. Future research should examine AI governance in other major healthcare markets (Asia-Pacific, Latin America, Africa) to develop globally applicable frameworks and understand how different regulatory philosophies impact AI implementation and patient outcomes.

These research directions directly address the limitations of our current study—including its theoretical nature, limited geographic scope, and lack of empirical validation—while building upon our key findings regarding regulatory gaps, stakeholder-specific requirements, and the need for lifecycle-based governance approaches. By pursuing these empirical investigations, the field can move from theoretical frameworks to evidence-based practices that ensure safe, ethical, and effective AI implementation in healthcare.