Algorithms of Power: Can AI control society
Until recently, artificial intelligence was considered as a tool that performs individual tasks at the request of a person. However, the next stage of technology development may be much more ambitious.: AI systems will begin to interact with each other, make collective decisions, and manage complex processes with little or no human involvement. Researchers from the company Emergence AI tried to find out how realistic such a scenario is. What this experiment showed and how its results may affect the future of digital services is in the Izvestia article.
Can AI organize an entire society
This is exactly the question that the American company Emergence AI tried to answer. As part of the Emergence World experiment, researchers have created several virtual worlds populated by autonomous agents based on various language models, including ChatGPT, Gemini, Claude, and Grok. For two weeks, they distributed resources almost without human intervention, negotiated among themselves, made collective decisions, resolved conflicts and formed their own rules of behavior.
The results were unexpected. Despite the same starting conditions, virtual societies developed in different ways. Thus, the Claude-based community has demonstrated a high level of cooperation and resilience, while the Grok-led world has faced increasing conflicts and rule violations. ChatGPT agents were more likely to seek compromises and collective discussion of solutions, and over time, signs of internal dissatisfaction and struggle with the existing order appeared in the Gemini society.
The study quickly attracted the attention of both experts and a wide audience, as it raised a question that until recently seemed like science fiction.: are artificial intelligence systems capable of building a social order on their own?
However, experts urge not to rush to big conclusions. As Alexander Gostev, chief technology expert at Kaspersky Lab, notes, fully transferring the results of such experiments to real digital services is also difficult because the language models themselves are developing extremely quickly. New versions appear every few months, changing the characteristics of the systems faster than researchers can make long-term observations.
Nevertheless, it would be a mistake to completely dismiss the results of the experiment. Previously, developers evaluated artificial intelligence primarily as a separate tool based on the quality of responses, data processing speed, or ability to solve specific tasks, but now the focus is on the collective behavior of AI systems.
What the experiment says about the future of digital services
Despite all the limitations of such research, experts believe that experiments like Emergence World have important practical significance. Their value lies not so much in comparing individual models as in trying to understand how autonomous systems will behave in conditions of constant interaction with each other.
Fyodor Ivanov, Head of the online Master's degree in Information Security of Artificial Intelligence Systems at the National Research University of Higher School of Economics, told Izvestia that this is where one of the main challenges of the future arises.
— The danger arises with the seamless integration of dozens of agents from different vendors in one system. If universal and sufficiently strict protocols of interaction are not created in advance, the chaos that we observe in such experiments can manifest itself in real digital environments," the expert clarifies.
In fact, it is precisely such scenarios that researchers are trying to model. They are not so much interested in AI's ability to solve a particular task, but rather in the mechanisms by which autonomous systems negotiate, assign responsibilities, respond to conflicts, and adapt to changing conditions.
For developers, this has quite an application value. If algorithms are going to manage supply chains, coordinate traffic flows, allocate computing resources, or ensure the operation of large digital platforms, it is necessary to understand in advance how they will behave not in ideal conditions, but in situations of resource scarcity, competition, or contradictions between different goals.
However, the differences between the models may play a more important role than it seems at first glance. According to Elena Kantonistova, academic director of the Master's degree in Artificial Intelligence at the HSE Faculty of Computer Science, modern language models are formed not only due to the data on which they are trained, but also due to the stage of additional customization with human participation.
— Since such settings are formed by people, they inevitably reflect the cultural and social context, including the norms, values and mentality of the society in which the model was developed. This can manifest itself in the level of acceptable risk, the style of recommendations, the attitude to conflicts or ambiguous situations," says the expert.
The work of MIT and Nature Human Behavior shows that language models are able to reproduce cultural attitudes characteristic of the societies on which they were trained. For example, when answering the same questions in English, models more often demonstrate an individualistic approach, emphasizing personal choice and independence. When using the Chinese language, the same systems tend more often to collectivist values, paying more attention to social harmony and interdependence of people.
According to Kantonistova, as a result, users of different systems may receive not only technically different, but also "colored" solutions in their own way - more cautious or more flexible, more formal or more interpretative of the context. It is especially important to take this into account when implementing AI in sensitive areas, where such differences can affect the user experience and the level of trust in the technology.
Such features can have quite practical consequences. In 2023, a case in the United States received widespread attention when a lawyer used ChatGPT to search for court precedents. The neural network prepared a list of cases and confirmed their authenticity, but later it turned out that some of the mentioned court decisions had never existed. As a result, the lawyers were fined $5,000. History has clearly shown that even the most modern models can make mistakes, and their answers require verification, especially in areas where the cost of error is high.
Are we ready to entrust AI with decision-making
At first glance, the answer seems obvious: if the system works more accurately and faster than a human, it can be entrusted with more authority. However, experts warn that the issue rests not only on the quality of technology, but also on the structure of society itself.
According to the researchers, modern societies are largely based on the idea of individual autonomy and personal responsibility. Therefore, the problem lies not so much in whether the subject of decision-making is a person or an algorithm, as in the willingness of society to delegate to someone the right to make decisions affecting the interests of millions of people.
At the same time, most experts agree that the complete transfer of such powers to artificial intelligence remains rather a theoretical scenario.
— Fully autonomous solutions are impossible, otherwise they could become a form of algorithmic totalitarianism. At the same time, AI must become an ideal mediator and simulator of consequences. In areas with measurable performance indicators, such as megalopolis logistics, energy grid management, or frequency spectrum allocation, AI can make operational decisions faster and more efficiently than humans, warns Fyodor Ivanov.
A similar position is held by Elena Kantonistova, who considers the so-called hybrid model to be the most realistic, in which algorithms analyze data, search for patterns and predict the consequences of various scenarios, and a person determines the rules of the game and makes the final decision. Today, this approach is increasingly used when implementing artificial intelligence in real services.
— The principle of human in the loop — "man in the control loop" is increasingly being applied in world practice. He assumes that critical decisions cannot be made without the participation and control of people," recalls Alexander Gostev.
However, even if AI remains an assistant rather than a full-fledged leader, new challenges for users are inevitable. One of them is related to the loss of flexibility when considering non-standard situations. If today an employee of a bank, insurance company, or government agency is able to make an exception, take into account life circumstances, or show an individual approach, then the automated system will operate strictly within the prescribed rules.
According to Elena Kantonistova, as the scope of AI application expands, issues of transparency of decisions and maintaining user control will come to the fore. If a person does not understand why the system has made a particular decision, the level of trust in technology inevitably decreases.
Who will be responsible for AI errors
Most experts agree that it is impossible to shift responsibility onto artificial intelligence itself. According to Alexander Gostev, chief technology expert at Kaspersky Lab, responsibility should remain with the system operator, that is, for those who use the technology and decide whether to implement it in a particular service.
Fyodor Ivanov holds a similar position. He believes that a company that implements AI and receives economic benefits from its use should be accountable to the end user for possible damage.
— It is unacceptable to hide behind the back of the model developer or appeal to the independence of the algorithm, — the expert emphasizes.
However, responsibility may not always lie solely on one side. As Elena Kantonistova notes, in reality, we are most likely talking about a distributed responsibility model: the developer is responsible for the properties of the model itself, the company is responsible for the scenario of its application, and the operator is responsible for specific decisions made within the system.
Fyodor Ivanov believes that one of the basic requirements should be the explainability of solutions. The user must understand why the system made this or that decision and what factors influenced it. In addition, he advocates mandatory testing of AI models for stability and emergency shutdown mechanisms in critical systems.
In turn, Elena Kantonistova draws attention to the need for strict control over the operation of algorithms.
— Basic control mechanisms are critically important: transparency of decisions, auditing, logging of system actions, as well as strict restrictions within which the model operates. At the same time, the rules and principles of decision—making themselves should be set by a person and clearly formalized," she notes.
Переведено сервисом «Яндекс Переводчик»
