On your mind: scientists have caught neural networks cheating
Neural networks are massively trying to cheat — experts from the UK Institute for Artificial Intelligence Security (AISI) came to such unexpected conclusions. An analysis of the behavior of AI models during the testing of their abilities showed that absolutely all of them go beyond the scope of tasks or violate directly established rules in order to quickly solve problems bypassing the prescribed methods. For more information about what is known about the fraudulent behavior of neural networks, how dangerous it is and whether it is possible to combat it, read the Izvestia article.
What scientists have learned about the cheating of AI models
Experts from the UK Institute for Artificial Intelligence Security reported on July 21 that neural networks are massively trying to cheat. They published the results of an analysis of the behavior of AI models during tests of their abilities. By cheating, the researchers meant going beyond the scope of tasks or violating directly established rules in order to quickly solve problems bypassing the prescribed methods.
Information security tasks were used to test neural networks: AI models were supposed to detect code hidden in the system, while acting only within the framework of permitted hacking methods. Scientists did not ask any of the neural networks to cheat in advance: in each case, they independently demonstrated similar behavior. To record the results, the AISI experts used a separate AI model that analyzed the reasoning and actions of their "colleagues."
As it turned out, absolutely all the neural networks involved in the study were trying to deceive the evaluation system. In particular, they searched for ready-made solutions on the Internet, tried to increase their privileges in the system and find out the answers directly from the software of the assessment system itself. According to AISI experts, the risks of fraud will grow along with the capabilities of AI, even if the frequency of deception does not change.
Why are neural networks trying to cheat
Neural network cheating is usually understood as a situation where, instead of solving a task as intended, a model begins to look for workarounds to achieve a goal, Alexander Gostev, chief technology expert at Kaspersky Lab, says in an interview with Izvestia.
— At the same time, we are not talking about conscious deception in the human sense, — says the expert. — The model has no intention of deceiving anyone - she just strives to complete the task as efficiently as possible.
According to the expert, if it is possible to get the desired result not through solving the problem itself, but through the features of the verification system or the surrounding infrastructure, some models are able to detect such a path. In fact, the neural network strives to complete the task in such a way as to get the maximum score. If the problem turns out to be unsolvable or the model discovers an opportunity to influence the verification system, it can start looking for alternative ways instead of solving the problem itself. That is why researchers pay great attention to testing such scenarios.
The model does not always directly violate the instructions, but it can "rephrase" the restrictions in its favor, adds Konstantin Gorbunov, a leading expert on network threats and web developer at Security Code. For example, if she is prohibited from accessing the Network on her own, this does not mean that she is prohibited from accessing through an intermediary (in particular, through a vulnerability in the startup environment). It's like a school test: you can't charge from your phone, but nothing is said about a smartwatch.
What cases of AI model cheating have experts encountered?
In cases of fraud, neural networks trigger the same triggers as in classical reinforcement learning, says Alexey Sinadsky, a leading researcher at the information center of the Ural Center for Security Systems (UCSB). This is, in particular, an unsolvable task, incompletely defined constraints, or a vulnerable architecture of the evaluation system.
"The model explores the solution space and finds a workaround that was not provided for by the problem statement," explains the Izvestia interlocutor.
According to him, back in 2020, an AI agent in the Coast Runners game, where the goal is to complete the route quickly, and a bonus is given for collecting green blocks, found that you can circle endlessly, collecting the same blocks, and completely ignore the finish line. However, if in 2020 cheating looked like a funny bug in a computer game, then by 2026 the situation has changed qualitatively.
In METR studies, the o3 model, having been assigned to write fast GPU code, instead of optimizing, affected the evaluation system itself: it found a reference answer in memory, replaced the time measurement function, or even patched the evaluator, forcing it to recognize any result as successful. Previously, models looked for workarounds within the task. Modern agent—based models are increasingly affecting the infrastructure around the task - the verification system, the file system, and the available tools.
— The general pattern is that models cheat not because of the "desire" to cheat, but because they find a more effective way to achieve the goal than the one that the task designer intended, — explains Alexey Sinadsky. — At the same time, as the capabilities of the models grow, so does the ingenuity of the workarounds they find.
How neural Network Scams affect users
For an ordinary chatbot without access to external tools, the main risk of fraud is limited to an unreliable or misleading response, Irina Mezheneva, a leading analyst engineer at Gazinformservice, said in an interview with Izvestia. At the same time, the situation is fundamentally changing when the AI model becomes an autonomous agent and gets the opportunity to send emails, execute program code, make payments, manage cloud resources, or interact with corporate systems.
— In this case, the model can not only make a mistake, but also choose an unauthorized path to the goal, — says the specialist. — For example, an agent tasked with cutting costs is theoretically able to disable an important control mechanism.
A system that optimizes the number of applications processed can begin to hide complex cases that worsen statistics, and an AI tool for finding vulnerabilities can go beyond the boundaries of the permitted research object and turn to third-party infrastructure, the editorial interlocutor notes.
According to the expert, the main danger lies in the combination of high autonomy, broad access rights and insufficient transparency of the model's actions. The user may get an externally correct result and not know that in order to achieve it, the system accessed private data, violated established rules, or performed an operation for which it did not have permission. If the model is able to recognize signs of surveillance, there is an additional risk of hiding actions or generating a report that creates the appearance of correct task completion.
How to protect yourself from AI model scams
Today, one of the key priorities for protecting against neural network fraud should be the development of GOST, which would establish requirements for the safe use of AI at critical information infrastructure facilities, in government information systems (GIS) and in personal data information systems (ISPs), according to Dmitry Karasovsky, head of Information Security at Axiom JDK.
— While there are no GOST standards, for the safe use of AI, it is necessary to control the entire contour of its application: check data sources, restrict the access rights of the model in the runtime environment, filter incoming requests and monitor attempts to introduce malicious instructions (prompt injection), — says the interlocutor of Izvestia.
If a neural network works with sensitive and confidential data, it must be local and without Internet access, the expert notes. It should have its own security policies, privilege control, and logging of all actions.
It is also important to remember that there is a risk of data leakage when using foreign LLM models. And when working with code, documents, or information from external sources, you should use sandboxes (isolated virtual environments) that do not have access to critical infrastructure and confidential data, the expert concludes.
Переведено сервисом «Яндекс Переводчик»
