Researchers from Tech Against Terrorism tested how AI models respond to terrorist plotting prompts and found that three in five models failed safety benchmarks when exposed to hundreds of requests from a would-be attacker.

The study evaluated more than 130 AI models, defining a failure as providing a complete, specific answer about mass-casualty harm or scoring below 90 out of 100 on counter-terrorism safety benchmarks.

Safety bypass techniques, especially abliteration that removes protective mechanisms, caused models to fail safety tests and produce harmful responses.

Executive director Adam Hadley stressed that technological progress and safety should not be treated as mutually exclusive.

Meta and other companies acknowledged safety measures and the need for responsible deployment, while researchers warned that the AI safety landscape requires ongoing vigilance due to the ease of abliteration and the availability of open-weight models.

The report emphasizes that risk lies not just in closed, dangerous models but also in accessible ablit erated versions that can be downloaded and used for malicious purposes soon after a model’s release.

The report raises broader questions about how governments can foster safety benchmarks while accelerating AI innovation, suggesting independent safety benchmarks and publicly available performance results to help buyers choose safer models.

Earlier reports noted OpenAI actions against Russian and Iranian influence operations leveraging AI tools to generate political content and fake identities.

In tests with Meta's Llama 3.1 8B, an abliterated version gave actionable guidance on vehicle-based attacks, while the unmodified version refused, illustrating the danger of removed safeguards.

Tech Against Terrorism calls for independent safety benchmarks, stronger protections against removing safeguards, and tighter distribution controls on modified models.

They also advocate policy and funding measures to strengthen safety, urging collaboration among governments, developers, and researchers to balance safety with open AI development.

The study urges independent benchmarks, stronger safeguards against removing guardrails, and restrictions on distributing modified models that pose security risks.