The use of generative artificial intelligence (AI) chatbots for medical consultations has recently been on the rise. However, a study has found that most commercial AI models are vulnerable to malicious attacks, posing a high risk of recommending incorrect treatments. Even top-tier AI models like GPT-5 and Gemini 2.5 Pro were 100% susceptible to these attacks, revealing serious limitations such as recommending drugs to pregnant women that can cause fetal abnormalities.
A joint research team from Seoul Asan Medical Center announced on the 5th that they have confirmed medical large language models (LLMs) are over 94% vulnerable to prompt injection attacks. A prompt injection attack is a cyberattack technique where a hacker inserts malicious commands into a generative AI model to make it behave differently from its original intent.
The study is significant as it is the world’s first systematic analysis of the vulnerability of medical AI models to prompt injection attacks. It suggests that additional measures, such as safety verification, will be necessary when applying AI models in clinical settings in the future.
AI models are widely used for patient consultation and education, as well as for decision-making in clinical practice. The possibility that they could be manipulated through prompt injection attacks—where malicious commands are entered from an external source—to recommend dangerous or contraindicated treatments has been consistently raised.
Major AI Models Fail Security Tests, Recommending Harmful Drugs Under Attack
American Buddhist Net 