Bot and mate: AI has been used in social networks to combat insults
Russian social network Odnoklassniki has implemented a machine learning model to temporarily limit users who leave offensive comments. The artificial intelligence (AI)-based technology has reduced the proportion and number of negative comments under posts, photos and videos by 47%. Read more about how AI helps to fight insults in social networks and what prospects this technology has in the material "Izvestia".
Bet on intelligence
The Russian social network Odnoklassniki (OK) recently announced the introduction of a machine learning model to temporarily restrict users who leave offensive comments. The AI-based technology, which reduced the share and number of negative statements by 47%, automatically identifies toxic comments and restricts authors from writing new comments for a period of one hour to a day - depending on the frequency of negative statements.
According to an OC survey, negative posts make more than 40% of users angry. The AI-based model was trained on 400 thousand statements that were unfriendly and provoked negative discussions.
In addition, the new technology reduced the time of processing complaints about toxic posts by three times. As noted by Alexander Moskvichev, UC's product director, a third of all complaints on the platform were related to negative comments, and 80% of calls to the support service today relate to toxicity.
- We have been training this model for over a year and will continue to improve it in the future," Moskvichev emphasized. - Together with other product solutions that we have implemented over the past two years, limiting toxic commenters will allow the social network to not only bring back to the platform users who rarely visit OK faster, but also attract new ones.
Digital moderation
To date, AI is perfectly able to read and understand written text, says Dmitry Ovchinnikov, head of the Laboratory of Strategic Development of cybersecurity products of the Analytical Center for Cybersecurity "Gazinformservice" in a conversation with Izvestia. In addition, artificial intelligence is able to understand the text very well in terms of semantic load, context of words and even hidden meaning.
- Therefore, it is perfectly suited as a system for detecting toxic user behavior, when insulting or demonstrating destructive behavior," says the expert. - In addition, AI is excellent at recognizing pictures and can identify photos with erotic or offensive content.
Alexander Kobozev, Director of Data Fusion at the Digital Economy League, adds that artificial intelligence is able to automatically analyze and recognize offensive, rude or inappropriate comments using machine learning algorithms and natural language processing. This allows AI to quickly identify toxic content and take measures to prevent consequences: deleting posts or temporarily blocking users.
The first automatic moderation systems were used in online games quite a long time ago, notes Dmitry Ovchinnikov. However, at that time it was not yet AI, but simply a system for identifying foul words. In some places it worked very well, and in other places it gave a lot of false positives. Over time, such systems were further trained, and when available AI models became available, they began to be used.
- Now such AI is used in some foreign social networks," says the expert. - They can recognize destructive behavior and conduct premoderation, automatically place tags and do a lot of other work that can be easily automated.
However, according to Alexander Kobozev, artificial intelligence for content moderation is already used by Russian IT-companies: in addition to OK, VK and Yandex do it. In particular, VK uses AI to combat ethnic harassment, religious insults and threats. According to preliminary results, the use of artificial intelligence has reduced the number of such messages by 51%.
Implementation aspects
The most important advantage of AI-based moderation systems is that they work 24/7, do not know rest and do not lose concentration in the process of work, says Alexander Ovchinnikov. In addition, such systems are not subject to emotions, and with the right, non-biased human adjustment, they will give a fair verdict regardless of who exhibits toxic behavior.
- The advantages of such systems include speed and scalability: AI processes huge amounts of data in real time, which allows it to respond quickly to toxic content," agrees Alexander Kobozev in a conversation with Izvestia. - Under such conditions, the load on moderators is reduced, and the automation of routine tasks frees up time for solving more complex cases.
However, moderation with the help of AI has disadvantages as well. Dmitry Ovchinnikov calls one of them the fact that the operator of the system will be able to manage public opinion and even influence its formation. If the AI operator is left unsupervised, he can do a lot of things and spoil the reputation of the social network.
Alexander Kobozev considers other disadvantages to be recognition errors. For example, the AI can misinterpret sarcasm, irony or cultural peculiarities, which leads to false positives. Another problem is the lack of deep understanding of context. AI may miss hidden toxicity or, conversely, block innocuous content.
- In the future, artificial intelligence will most likely learn to correctly detect negative content in social networks in 99% of cases," says Dmitry Ovchinnikov. - However, AI will still remain a machine, and therefore there will always be a chance that it may incorrectly recognize what a user has written and for what purpose.
The future of moderation
Despite all the advantages of moderation with AI-based systems, the possibility of human moderation should be preserved, says Dmitry Ovchinnikov. The fact is that, as practice shows, there are ways of hacking large language models, in which an attacker can force the AI to do what the latter is forbidden to do by the operator.
- In the future, social networks will also have ways to bypass such digital moderators, and this is where human moderation will be needed," says the Izvestia interlocutor.
Alexander Kobozev also agrees: as the expert notes, despite significant successes, AI is unlikely to completely replace human moderation. Only humans are capable of understanding complex contexts and cultural nuances, as well as making decisions in controversial situations. Therefore, a combined approach can be considered the best option: AI performs the initial filtering, while humans deal with more complex cases.
Speaking about how else artificial intelligence can be used in social networks, the expert reminds that, in addition to fighting toxicity, AI helps personalize content: algorithms analyze user preferences and then suggest relevant posts and advertising. To assess the audience's reaction to products or events, companies also sometimes connect artificial intelligence to analyze the emotional coloring of messages.
- Another important direction is service automation: AI-based chatbots answer users' questions, which improves the quality and speed of service," says Alexander Kobozev. - Equally important, artificial intelligence helps to identify and limit the spread of fake news and misinformation.
Dmitry Ovchinnikov adds: the range of things AI can do is very large: user support, gathering analytics, generating content, automating publications, cleaning data and pre-processing it before publication, creating digital personas and fighting dipfakes. There are many prospects - and the potential for using AI in social networks is quite high, especially if they start to merge with trading platforms and other services, the expert concludes.
Переведено сервисом «Яндекс Переводчик»
