Why the FDA Needs to Increase Transparency in AI-Powered Medical Devices

When used in healthcare, artificial intelligence (AI) promises to improve patient outcomes, reduce costs and advance medical research. These tools can analyze patient images for disease, detect patterns in large health datasets, and automate certain administrative tasks. But many companies are developing AI-enabled medical products in what is essentially a “Black Boxrevealing little to the public about their inner workings. Just as doctors and patients need to know what’s in a prescription drug, AI users need information about tools that can be used to help make life-and-death medical decisions.

Not all AI-enabled tools fall under the jurisdiction of the Food and Drug Administration, but the agency regulates any software intended to treat, diagnose, cure, mitigate, or prevent diseases or other conditions before it is used. cannot be traded and sold commercially. In recent years, the FDA has considered a updated approach oversight of these products, including steps to improve how developers communicate about four key factors: the intended use of a product, how it was developed, its performance and the logic it uses to generate a result or recommendation.

If companies do not disclose these details, prescribers and patients may be more likely to use products inappropriately, which can lead to inaccurate diagnoses, inappropriate treatment and harm. Here is how and why this information is important for patients and prescribers:

  • Intended use. AI developers should clearly communicate how their products should be used, such as specifying the exact target populations and clinical parameters, as these factors can greatly affect their accuracy. For example, Mayo Clinic researchers developed an AI-based tool to predict atrial fibrillation using data from the general population receiving care at the facility. Although it is very accurate when used on this general population, it works only slightly better than chance in high-risk clinical scenarios, such as on patients who have just had some type of heart surgery.
  • Development. Clinicians need information about the data used to develop and train AI systems to better determine if and how to use certain tools for specific patients. If the data comes from a limited population, for example, the product may incorrectly detect or miss a disease in people who are underrepresented (or not represented at all) in the training data collected. For example, AI-based smartphone apps designed to detect skin cancer can often be trained on images of most lighter-skinned patients. As a result, the products may not work as well on patients with darker skin, which may lead to inappropriate treatment and the potential to exacerbate existing health disparities.
  • Performance. Prescribers and patients should know whether AI tools have been independently validated and, if so, how they were evaluated and how well they performed. Currently, this information may be difficult to obtain and compare across tools as there are no set standards on how these products should be evaluated and not independent organization to ensure their proper use. In one case, researchers at a hospital system found that an AI tool developed to predict sepsis missed two-thirds of cases and was associated with a high rate of false alarms. the developer claimed, however, that “the researchers’ analysis did not take into account the required tuning that should precede the actual deployment of the tool”. Performance issues also arise when AI developers use the same data to train and validate their products. This can lead to inflated accuracy ratessimilar to students using the same test for practice and the final exam.
  • Logic. Some AI tools, especially those enabled by machine learning techniques, are called black box models because how they arrived at a result or recommendation cannot be explained. In other cases, a developer may keep this type of information confidential. However, if clinicians and researchers are unable to understand the logic used by a tool to reach its conclusion, they might not trust the recommendations it makes or be able to identify potential flaws or limitations of its performance. For example, an AI model used to analyze X-ray images made predictions based in part on the type of equipment used to take the image, rather than the actual content of the image. If the logic of the model had been more transparent at the start, this defect could have been corrected earlier.

The FDA can promote increased transparency requiring more and better information about AI-enabled tools in the agency’s public database of approvals. Currently, the details companies publicly report about their products vary. For example, in a To analyse of the public summaries of the 10 FDA-approved AI products for breast imaging, only one provided information on the racial demographics of the data used to validate the product. Requiring developers to publicly report basic demographic information and, where available, product performance data in key subgroups, could help providers and patients select the most appropriate products. This is especially important when treating conditions that have disparate impacts on underserved populations, such as breast cancer, a disease more likely to be fatal for Black woman.

Similar to its drug labeling requirements, the agency could also require developers to provide more detailed information on product labels so these tools can be properly evaluated before being purchased by drug establishments. health or patients. Researchers from duke university and the Mayo Clinic suggested a nutrition label-like approach that would describe how an AI tool was developed and tested and how it should be used. This would allow end users to better evaluate products before they are used on patients. The information could also be integrated into a facility’s electronic health record system to facilitate data access for busy providers at the point of care.

AI can save lives and reduce healthcare costs, but providers and patients need to know more about these products to use them safely and effectively. The FDA should continue its crucial work to increase the transparency of these revolutionary tools.

Liz Richardson leads The Pew Charitable Trusts Healthcare Products Project.

Comments are closed.