Yandex assessed the economic effect of increasing the efficiency of the computing infrastructure

Over the past six months, Yandex has managed to save 9.2 billion rubles due to engineering optimization of the computing infrastructure. The company has allocated the released resources to scale developments in the field of artificial intelligence, its press service reported.

One of the key areas of the optimization program is the dynamic redistribution of computing resources. When the user load decreases, the available capacity is automatically redistributed to other tasks. For example, at night, when user activity decreases and the load on services decreases, graphics processing units (GPUs) are automatically connected to the training of new models. This practically eliminates the downtime of expensive computing resources.

Additional savings were provided by optimizing the models themselves. Some of the models were transferred to the MoE ("Mixture of Experts") architecture, which uses only a part of the neural network to process each request.

The engineers also used an approach in which a small model is trained based on a larger model that has already been trained, as well as quantization, which allows calculations to be performed in a more efficient format. At the same time, the company accelerated the loading of data for model training, parallelized calculations, implemented caching of repetitive calculations and EAGLE technology, which allows faster generation of answers by predicting the next few tokens.

The program covered not only the infrastructure for artificial intelligence, but also the computing power that powers the rest of the company's services. Specialists have improved the operation of servers with central processing units (CPUs), which provide search, selection of recommendations and other resource-intensive tasks. To do this, they used PGO and BOLT compilation optimization technologies, which analyze the real load of services and help processors perform more operations without increasing the resources used due to a more accurate organization of the program code.

In addition, engineers have reviewed the rules for storing data at all stages of its lifecycle. To do this, storage rules have been optimized, excessive data duplication has been eliminated, new compression algorithms have been tested, and tools have been introduced that automatically find unused tables, fields, and backups, as well as help distribute data more efficiently between different types of storage (HDD, SSD, and others). As a result, the load on the infrastructure has been reduced, and a single process of constant search and elimination of "losses" of computing resources has been built.

According to Yandex, the company regularly conducts technical and business audits of computing power usage in order to increase the efficiency of the existing infrastructure and direct the released resources to the development of artificial intelligence technologies and new products.

The program is implemented by engineers from the Search Services and AI business group together with the Yandex internal infrastructure team. Many changes were implemented in all products at once thanks to a single ML platform, a set of technologies that covers the entire cycle of working with models: from training to operation. This allows for faster implementation of engineering solutions and more efficient use of existing infrastructure, the company stressed.