In its official documentation, Microsoft has acknowledged that while the new generation of AI agents integrated into Windows 11 are poised to become pivotal features for the future and their deployment is steadily progressing, they also present potential security risks. These intelligent agents are designed to mimic human actions and execute complex, multi-step tasks. However, due to inherent limitations in their underlying models, they may occasionally produce hallucinations—that is, generate outputs that are inaccurate or irrelevant. Additionally, they are susceptible to novel threats, such as cross-prompt injection attacks, where malicious actors can manipulate instructions to cause data breaches or facilitate the installation of malware.

To mitigate these risks, Microsoft has opted to disable the AI agent feature by default, requiring users to manually activate it. The company also advises users to conduct a thorough risk assessment before enabling the feature. Despite these concerns, Microsoft remains committed to advancing AI integration across its platform, as evidenced by innovations like Copilot Voice and Vision. These enhancements aim to elevate user experience and productivity, though striking a balance between innovation and security continues to pose a significant challenge.