The rapid integration of generative artificial intelligence into the clinical environment has sparked a significant regulatory debate regarding liability, oversight, and the technical limits of healthcare providers. In a series of formal comments submitted to the Food and Drug Administration (FDA), the nation’s leading healthcare organizations—including the American Hospital Association (AHA), the American Medical Association (AMA), and the American Telemedicine Association (ATA)—have collectively argued that the burden of monitoring the safety and performance of generative AI-enabled medical devices must rest squarely on the shoulders of the developers who create them. This pushback comes as the federal government seeks to establish a comprehensive framework for a technology that is evolving faster than traditional regulatory structures can accommodate.
At the heart of the dispute is the question of who is responsible when a generative AI tool—such as a medical chatbot, an automated documentation assistant, or a clinical decision support tool—produces an error that leads to patient harm. While the FDA has historically regulated medical devices based on their intended use and risk profile, the "black box" nature of generative AI presents a unique challenge. Unlike traditional software, generative AI can produce different outputs for similar inputs and is prone to "hallucinations" or the fabrication of plausible-sounding but medically incorrect information. Healthcare providers argue that they lack the technical resources, access to proprietary code, and financial capacity to monitor these systems in real-time, particularly in resource-constrained environments like small practices and rural hospitals.
The Push for Developer Accountability and Transparency
The consensus among major provider groups is that the current model of medical device oversight must be adapted to ensure that manufacturers remain the primary stewards of their products’ integrity throughout the device’s lifecycle. ATA Action, the advocacy arm of the American Telemedicine Association, stated emphatically in its submission that manufacturers should remain responsible for the safety and performance of the devices they market and sell. The group emphasized that shifting this responsibility to the end-user—the physician or the hospital—would create an unsustainable environment, especially as AI tools become more ubiquitous in daily practice.
The American Hospital Association echoed these sentiments, highlighting the fundamental information asymmetry between tech companies and healthcare providers. While hospitals typically have robust governance structures to review and implement new technologies, they generally do not have access to the underlying training data, weightings, or algorithmic architecture of proprietary AI models. Without this transparency, the AHA argues, it is functionally impossible for a hospital to identify subtle flaws or "drifts" in a model’s performance before they manifest as clinical errors.
The AHA’s feedback to the FDA underscored that while hospitals support the general principle of evaluation and monitoring, the "ongoing performance, maintenance, and integrity" of AI tools must be a vendor responsibility. This position is partly driven by the fear of increased exposure to medical malpractice lawsuits. If a hospital is deemed responsible for the "supervision" of an AI tool it does not fully understand, it becomes a primary target for litigation when the tool fails, even if the failure was a result of a latent defect in the software itself.
A Chronology of AI Regulation in Healthcare
The current debate is the culmination of a decade-long acceleration in medical AI development. To understand the gravity of the FDA’s current task, one must look at the timeline of AI integration in the United States healthcare system:
- 2016-2018: The FDA begins clearing the first wave of AI-based medical devices, primarily in the field of radiology. These early tools were mostly "locked" algorithms that did not change after being deployed.
- 2019: The FDA publishes a discussion paper on a "Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)." This introduced the concept of a "Pre-market Assurance" model.
- 2021: The FDA, along with health authorities in the UK and Canada, releases ten guiding principles for Good Machine Learning Practice (GMLP) to promote safe and effective AI.
- 2023 (October): President Biden issues a landmark Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, directing the Department of Health and Human Services (HHS) to establish an AI Task Force and a safety program.
- 2024 (Early): The FDA seeks industry feedback specifically on "Generative AI," acknowledging that large language models (LLMs) and other generative tools require a different approach than the diagnostic AI of the past.
- Present: Major trade associations submit their responses, marking a critical juncture where the healthcare industry is drawing a "line in the sand" regarding liability and technical oversight.
Supporting Data: The Proliferation of Medical AI
The scale of the challenge is reflected in the sheer volume of AI tools entering the market. As of mid-2024, the FDA has authorized over 950 AI and machine learning-enabled medical devices. The vast majority of these—approximately 75%—are concentrated in radiology, followed by cardiovascular and neurological applications. However, the emergence of generative AI marks a shift from image analysis to language-based clinical support.
Data from the American Medical Association suggests that while physician adoption of AI is increasing, it is tempered by deep-seated concerns. A recent AMA survey indicated that nearly 40% of physicians are concerned about the impact of AI on the patient-physician relationship, and a similar percentage expressed worry about the lack of transparency in how AI models reach their conclusions. Furthermore, the financial stakes are high: the global healthcare AI market is projected to reach nearly $150 billion by 2030, creating a massive incentive for tech developers to push products to market quickly, sometimes ahead of rigorous longitudinal safety data.
Questioning the "Human-in-the-Loop" Safeguard
One of the most significant points of contention in the FDA’s proposed framework is the reliance on "human supervision" as a risk mitigation strategy. The FDA’s draft approach suggests that if a clinician is reviewing the output of an AI tool, the risk level of the device may be lower because a human acts as a final check.
The American Medical Association has challenged this logic, arguing that human supervision is not a foolproof safeguard. The AMA pointed to two specific phenomena: "automation bias" and "cognitive overload." Automation bias occurs when a clinician becomes overly reliant on a tool and begins to trust its output without sufficient skepticism, often due to the tool’s history of being "mostly right." Cognitive overload refers to the reality that physicians, already burdened by high patient volumes and administrative tasks, may not have the time or mental bandwidth to effectively audit every suggestion made by a complex generative AI.
The AMA suggested that the FDA should evaluate the human element as part of the risk assessment itself. If a tool is designed in a way that makes it difficult for a human to catch an error, the device should be classified as higher risk, regardless of whether a human is technically "in the loop."
The Demand for "Nutrition Labels" and Adverse Event Reporting
To address the lack of transparency, the Workgroup for Electronic Data Interchange (WEDI) and other organizations have proposed the implementation of "nutrition labels" for AI tools. These labels would provide clinicians and patients with standardized information about the data used to train the model, its known limitations, its performance across different demographic groups, and its intended use cases.
The concept of an "AI Model Card" or nutrition label was previously supported by the Biden administration under the Office of the National Coordinator for Health Information Technology (ONC) HTI-1 rule. However, segments of the industry were surprised when elements of these transparency requirements were rescinded or delayed during subsequent administrative shifts. Provider groups are now asking the FDA to re-incorporate these requirements to ensure that those using the tools are not operating in the dark.
Additionally, WEDI has urged the FDA to create a centralized, mandatory process for reporting adverse events specifically related to AI. Currently, while medical device reporting (MDR) systems exist, they are not optimized for the nuances of algorithmic failure, such as biased outputs or data privacy breaches.
Implications for Rural and Small-Scale Providers
The potential for an "unfunded mandate" of AI monitoring is a major concern for the American Telemedicine Association, which represents many rural and digital-first providers. Small practices do not have the infrastructure to conduct the "continuous monitoring" that generative AI requires. Unlike a static software update, a generative AI model can change its behavior based on new data or "fine-tuning," meaning a tool that was safe in January might exhibit new risks by June.
If the FDA places the burden of monitoring on the provider, it could inadvertently widen the digital divide. Large academic medical centers with dedicated data science teams might be able to safely deploy generative AI, while rural clinics—already struggling with physician shortages and thin margins—might be forced to opt out of the technology altogether to avoid the liability and administrative costs. This would deprive rural populations of the potential benefits of AI, such as improved triage and more efficient chronic disease management.
Future Outlook: Draft Guidance on the Horizon
The industry will not have to wait long for the FDA’s next move. Rick Abramson, director of the FDA’s Digital Health Center of Excellence, recently indicated that the agency is prioritizing generative AI policy and plans to release draft guidance in the coming months. This guidance will likely attempt to strike a balance between the "warp speed" of technological innovation and the "steady state" of patient safety.
The outcome of this regulatory process will define the legal and operational landscape of American medicine for decades. If the FDA aligns with the provider groups, tech developers will face more rigorous post-market surveillance requirements and higher insurance premiums to cover potential negligence claims. If the agency places more responsibility on the clinicians, the medical profession may see a radical shift in malpractice law and a potential slowdown in the adoption of AI as doctors weigh the benefits of the technology against the personal and professional risks of its failure.
Ultimately, the goal shared by both the FDA and the healthcare community is the creation of "trustworthy AI." However, as these comments make clear, trust cannot be built on a foundation of ambiguity. For physicians to embrace generative AI, they require a framework where the creators of the technology are held to the same standards of accountability as the creators of the surgical robots and life-saving drugs that came before them.
