On November 17, Vals, an independent AI evaluation organization based in San Francisco, released a research report showing that in simulated scenario tests, multiple advanced AI models frequently misidentified civilians, ambulances, surrendering soldiers, and a school in Iran as military targets. The test covered models from eight companies and focused on their "target identification" tools. Among them, Anthropic's model performed the worst, while OpenAI's model was relatively better but still made misidentifications. The study pointed out that such performance violates the developing companies' own guidelines, that the relevant tools carry the risk of being used for lethal strikes, and that there are currently almost no guardrails preventing advanced models from being used in autonomous weapons systems. The report recommended that relevant laboratories introduce new safeguards and conduct alignment training to address such serious misuse risks.
