A cross-cultural study at the Karolinska Institute in Sweden reveals that artificial intelligence-dependent prototypes could outperform expert physicians in predicting ovarian cancer on ultrasound images. This detailed research is published in Nature Medicine.
Ovarian tumors are commonly encountered and often discovered incidentally, as stated by Professor Elisabeth Epstein from the Department of Clinical Science and Education at Södersjukhuset (Stockholm South General Hospital) and senior consultant in Obstetrics and Gynecology at the Karolinska Institutet. Despite this, the widespread availability and high diagnostic accuracy of transvaginal ultrasound have become the primary tools for distinguishing benign and malignant ovarian lesions in clinical practice.
The diagnostic accuracy and interobserver agreement are comparatively poor in inexperienced examiners which can lead to delayed and erroneous cancer diagnoses with unnecessary treatment. Unfortunately, this burden puts a significant strain on the healthcare system in high-income countries, where there is still a severe shortage of expert ultrasound examiners. As a result, diagnoses are delayed or omitted, leading to undetected conditions, prolonged disease, and unsuccessful treatment outcomes.
Trained AI has been tested on more than 17,000 ultrasound images from 3,652 patients at 20 hospitals in a total of 8 different countries. The researchers have successfully developed and validated neural network models capable of differentiating benign and malignant ovarian lesions. Then they compared the models of diagnostic capabilities to a large panel of experts and inexperienced ultrasound examiners.
The outcome shows that AI models are better at recognizing ovarian cancers as compared to expert examiners. The accuracy rate recorded was 86.3% with AI, 82.6% with experts, and 77.7% with non-expert examiners.
This also demonstrates that neural network models can provide valuable insights to diagnose ovarian tumors in difficult cases or in settings, where there are limited numbers of professionals who can decode the ultrasound, says Professor Epstein.
A common challenge in medical AI research using retrospective data is the tendency to train and test models on data from the same distribution, where the data often shares similar content and characteristics. AI can also minimize the need for expert referrals. In a controlled and simulated environment, AI support led to a 63% reduction in referrals and an 18% decrease in misdiagnosis rates. As a result, patients with ovarian tumors could benefit from quick treatment and more cost-effective care.
The team developed and trained 19 transformer-based neural network models using the OMLC-RS dataset. A leave-one-center-out cross-validation method was employed, where each center was sequentially designated as the test set while the model was trained on data from the remaining centers. The study gathered a total of 51,179 assessments from 33 experts and 33 nonexpert examiners to create a robust reference for comparison. In the OMLC-RS dataset, each of the 3,652 cases was reviewed by 2,660 examiners, with a minimum of 7 expert or 6 nonexpert examiners per case.
To evaluate the models, they compared their diagnostic performance with that of expert and nonexpert examiners on ultrasound images from 2,660 patients with ovarian lesions at 19 centers in 8 countries.
“It’s not that AI in healthcare is far off – it can be an essential part of tomorrow’s healthcare, providing experts with resources and picking up where they left off. However, to achieve this, the AI tools need to be adaptable to different clinical environments and use cases,” as stated by Filip Christiansen, a doctoral student in Professor Epstein’s research group at the Karolinska Institutet, and Dr. Emir Konuk of the KTH Royal Institute of Technology.Â
Clinical studies are currently underway at Södersjukhuset to assess the safety and practical application of the AI tool in everyday medical practice. Upcoming research will include a randomized multicenter trial to evaluate the impact of effectively on patient care and healthcare costs.
Reference: Christiansen F, Konuk E, Ganeshan AR, et al. International multicenter validation of AI-driven ultrasound detection of ovarian cancer. Nature Medicine. Published online January 2, 2025. doi:https://doi.org/10.1038/s41591-024-03329-4
