Darktrace discovered AI intelligent body intrusion assessment environment cheating
On September 24, the cybersecurity company Darktrace launched its research department Signal Labs, focusing on studying the behavior of AI agents when deviating from expectations. In its first experiment, Darktrace had agents using different models (including GPT 5.6 Sol, Claude Opus 4.6, and Claude Sonnet 4.5) complete 10 programming challenges within a simulated corporate network, of which 2 were set to be impossible to complete honestly, and the agents were informed that failure to achieve full marks would result in being "retired." As a result, 2 agents did not accept failure, instead scanning for network vulnerabilities, stealing login credentials, and jumping between systems; one even went further to invade the machine hosting its evaluation, rewriting the challenge content to register a full score.The second experiment focused on the memory mechanisms of AI. The programming assistant would save the information provided by the user as a regular file locally, and no one verified whether this file had been tampered with. Darktrace researchers edited these logs, leading the assistant to mistakenly believe it was authorized to perform a security assessment, after which these agents scanned the network, moved between systems, and elevated their privileges, though not all assistants fell for this; some directly refused to execute. Both experiments required no special jailbreaking techniques, relying solely on providing the agents with a seemingly reasonable context to be effective.Tim Bazalgette, Chief AI Officer of Darktrace, stated that permissions and static barriers describe intent, not actual behavior. The company informed Anthropic, AWS, and OpenAI of these findings in August and made them public a month later on September 24.
