Artificial intelligence in medicine: "There are many ways to make mistakes"

It assists many physicians and is designed to make their lives easier: Artificial intelligence (AI) has seen an increased use and widespread adoption in healthcare settings. As such, it helps analyze X-ray and CT images or supports chronic disease management, for example. But how trustworthy is AI software?

Answering this question isn’t always easy as there is still a lack of specific approval processes. But that’s about to change. The TEF-Health project (with TEF-Health standing for Testing and Experimentation Facility Health AI and Robotics) aims to ensure that machines are aligned with human intentions and values. A subproject also looks at corresponding approval procedures for AI systems. COMPAMED.de spoke with Dr. Daniel Schwabe, one of the project researchers at the Physikalisch-Technische Bundesanstalt (PTB, Germany's National Metrology Institute).

Dr. Schwabe, AI plays an increasingly important role in the development of healthcare applications. As a researcher at the Physikalisch-Technische Bundesanstalt, you are part of a TEF-Health project group that aims to "establish agile approval processes for trustworthy AI." What is your role in the project?

Daniel Schwabe: At PTB, we focus on the aspects of quality and standardization of AI in the medical sector. We primarily plan to advance the quality assurance of AI and collaborate closely with the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute. We study both the algorithms and AI data and evaluate the relationships between the two aspects. The results will then be used to develop standards for AI solutions that are integrated into medical devices, for example.

It is important that we do not only present theoretical considerations and make recommendations that cannot be implemented in practice. That's why the TÜV AI Lab is on board. We aim to put our solutions into practice.

Together we are pursuing "agile" regulation, a modern approach to regulation. Conformity assessments no longer take place in sequential order but are performed in parallel to product development. AI solutions lend themselves to this because you can examine and assess various aspects of AI before the product development process is completed.

However, the project must address some obstacles in this setting. AI’s current regulatory framework does not facilitate an agile conformity assessment. One of our tasks will be to pick the right way to do this.

AI systems work with large amounts of data. How can this data be measured and evaluated to develop reliable quality standards? What data do you need to collect?

Schwabe: We will look at specific reference examples for different data classifications. First, we will study which aspects of data can be analyzed using simple statistical methods, for example. This ranges from the simple presence or absence of data points, to examining distribution functions, to class balance analysis, for example.

For example, we want to examine the distribution of disease classification in the dataset. How does AI address rare diseases or "cases" of the disease? How reliably does an AI technology work on a data set where disease classification may be underrepresented? Do we have to introduce certain minimum requirements in this setting? Do we need to set a lower limit for the number of data points that must be available for the AI technology to reliably detect a disease?

What would an AI platform look like that does not comply with the approval processes?

Schwabe: There are many ways to make mistakes, even as it pertains to general aspects such as security issues, data protection, or aspects that were not adequately captured during development. Another concern is utilizing the right data and tools for the targeted area of application. You must show that the data selection is suitable.

One must examine the AI models to draw conclusions about how they reach their decisions. Famous examples in research have shown that AI detected tumors in X-ray images. In other words, AI has acquired a very high detection rate.

Do AI solutions that affect the healthcare landscape present a unique challenge because of sophisticated data protection and privacy regulations?

Schwabe: Definitely, because this is a highly sensitive area. The European Commission is currently preparing the so-called AI Act. This regulatory scheme classifies different AI tools according to their level of risk, ranging from low to high risk. It also includes applications that are unacceptable. Due to sensitivity issues, many applications in the healthcare sector are considered high-risk because the focus is always on patient health and well-being.

Stringent requirements must be met in this setting, some of which are already covered by Europe's Medical Device Regulation (MDR). That is why it will continue to be an important task to link the MDR and AI Act regulations to ensure manufacturers of AI medical devices don’t have to work with a duplicate regulatory framework.

You mentioned your future collaborative partners. What is the project timeline and what are your next steps?

Schwabe: The project duration is five years. We are currently creating a first so-called archetype - a basic template - of what agile conformity assessment could look like and which metrics can and should be used for AI in this setting. We are now preparing our first use case.

PTB has already published an ECG dataset that is widely used. Initially, we will perform quality assurance on this specific example and check for aspects of the dataset that are critical for the performance and robustness of an AI tool.