Oil painting: a new algorithm will allow you to quickly identify fake images on the Web

A new method for image analysis has been proposed in Russia. It helps to identify inconsistencies between individual elements of illustrations and, in general, contradictions to common sense. The development is based on an AI algorithm that works with text descriptions of images. The proposed approach increases the accuracy of visual data analysis and reduces computational costs. The development will find application in algorithms for detecting fake photos, interpreting satellite images, machine reading handwritten texts and other applications, experts believe.

How to recognize images

Russian scientists have developed an original and cost-effective approach to identify internal inconsistencies in images. In particular, it helps to quickly calculate visual data that contradicts common sense. Scientists from the AI Institute, Skoltech, the MWS AI Competence Center for Artificial Intelligence and the Moscow Institute of Physics and Technology participated in the development.

— Imagine a picture of a medieval knight with a mobile phone in his hand or a penguin on a bicycle. In such cases, a person instantly realizes that although each individual object looks normal, "there is something wrong with such images in general." Our brain automatically compares what it sees with the database of accumulated knowledge about the world and identifies inconsistencies. This task is much more difficult for artificial intelligence," Alexander Panchenko, PhD, one of the developers, head of the Computational Semantics group at AI and the NLP group at Skoltech, told Izvestia.

Photo: AIRI press service

He added that the existing systems recognize individual objects well. For example, a knight, a phone, a bicycle, a penguin. But they hardly understand whether these elements are compatible with each other from the point of view of common sense. In solving this problem, the researchers proposed using AI algorithms to analyze not the image itself, but its text description.

The first stage involves using AI to create simple facts about individual elements of an image, meaning the program literally lists what it sees. However, in order to describe the picture from different angles and not repeat itself, she chooses statements that are as different from each other as possible, the scientist explained.

The chip solution: How is the modern AI race unfolding?

Moscow relies on cooperation with the BRICS countries and China

Then, a special neural network language model converts phrases into numerical sequences — vectors. At the same time, similar proposals receive similar vectors. This allows you to mathematically compare the meaning of different statements about an image.

This method of data processing requires much less computational resources than visual image processing algorithms, which are trained on huge amounts of pre-marked information.

Photo: AIRI press service

At the last stage, the system automatically compares numeric vectors, and if it finds completely different ones, it concludes that the image is strange or contradictory. If there are no strong differences, the picture is considered normal.

— Thus, if a strange image is submitted to the input, the system begins to describe it in contradictory phrases. For example, for a picture with a knight, there is a discrepancy in the statement "the knight is holding the phone". It helps to identify the strangeness of the image. In the future, it only remains to fix this, which can be done, for example, using a contradiction classifier, a semantic sequence model, and some other methods," explained Alexander Panchenko.

Photo: AIRI press service

According to him, the proposed method has been tested on real image libraries that researchers use to test programs. Its accuracy proved to be 0.5–15% higher than other known models, depending on the data set. At the same time, the new approach turned out to be much more economical.

Hacking code: creating software using AI increases the vulnerability of companies by 15 times

Billions of rubles worth of business damage in the Russian Federation is also caused by deepfakes.

Where AI image analysis methods are in demand

The development opens up opportunities for creating more reliable computer vision systems, and the approach can also be applied in content moderation structures. For example, after refinement and further training on the relevant data, the program will be able to learn how to identify fakes — fabricated photos that reflect facts that do not correspond to reality, Alexander Panchenko added.

— The modern world is drowning in content. With this volume of visual information content, with the development of AI image generation technologies and the creation of fakes, people can no longer be sure that they are seeing reality. Unfortunately, such images are becoming the new norm. Therefore, the creation of such algorithms is not just a scientific race, but also an extremely important issue of trust in information," Anna Pyataeva, head of the scientific and educational laboratory of Artificial Intelligence Systems, associate professor at Siberian Federal University, told Izvestia.

Photo: AIRI press service

In the presented work, she noted, there is an obvious shift from just "recognizing objects" to understanding the meaning of an image. This is the moment when AI gets closer to human perception. He begins not only to see, but to understand what is wrong. In addition to the obvious applications, such as content moderation and image authentication, development may be in demand in industrial lines as it develops. For example, to check product quality, monitor the environment using satellite images, as well as in systems for recognizing manuscripts and ancient documents.

In addition, the proposed approach can be useful in synthetic image quality assessment systems that reproduce the statistical characteristics of real paintings, but do not correspond to specific objects, events or people, added Maxim Mitrokhin, Head of the Computing Engineering Department at Penza State University, Doctor of Technical Sciences. Such data is widely used in machine learning algorithms.

— Searching for "oddities" in a photo or video is one of the ways to understand that the material was created using artificial intelligence. At the same time, the industry remains open to the question of how and whether AI—edited images should be recognized, as well as what percentage of intervention is considered sufficient to recognize the material as generated," said Tatiana Deshkina, Deputy Director of Products at VisionLabs.

Photo: IZVESTIA/Yulia Mayorova

In her opinion, in the near future, creative products that are created without the use of AI will probably begin to be labeled with a special sign, similar to "non-GMO" products. At the same time, it will be necessary to develop certain standards and regulations for such identification in order to avoid misleading consumers.

— Processing text information requires less computing power on average than image analysis. Therefore, it is really possible to check images for realism by translating their meaning into text," said Petr Ermakov, ML Brand Director at Yandex.

However, it is important to keep in mind that encoding images into text is irreversible. Even the most detailed textual description cannot fully convey visual information, leaving room for various interpretations, the expert concluded.