In controlled tests, top language models correctly identified conditions in about 94.9% of predefined scenarios but chose the correct action in only 56.3% without human participants.

A new Oxford study finds AI health tools often give inconsistent or inaccurate medical guidance, raising risks for users seeking medical advice.

When real users or doctors are involved, AI's ability to translate diagnosis into safe, appropriate treatment falls short, highlighting a gap between recognition and guidance.

Independent voices outside the study emphasize caution, underscoring the need for careful use of AI in medical contexts.

In full-text scenario testing, AI correctly identified conditions in 94.9% of cases, showing strong diagnostic recognition but not necessarily appropriate action.

Experts advocate ongoing improvements with clear national regulations, guardrails, and medical guidelines to govern health-related AI use.

Some experts argue newer healthcare-focused AI systems could perform better under strict regulatory and professional recommendations.

Despite risks, there is acknowledgment of potential AI roles in healthcare if safeguards and health-specific versions with improved safety are implemented.

Many specialists see promise in healthcare-focused AI with regulatory safeguards and national guidelines to improve safety.

The study received funding from Prolific, the Dieter Schwarz Stiftung, and UK/US government bodies, with plans for broader, multilingual follow-ups.

Researchers plan cross-country studies to see if outcomes vary by context, continuing support from the same funders.

Authors intend to broaden the scope of research, building on initial findings with further development.