Just a gesture: scientists have simplified robot control in extreme conditions

Russian scientists have created a new technology for contactless control of industrial robot manipulators using hand gestures of the operator. Now the operator will not need to use remotes, keyboards, joysticks or special gloves with sensors. The technology opens up possibilities for remote control of machines in dangerous environments where human presence is impossible. Read more about the innovation in the Izvestia article.

Contactless interface for robot control

Sirius University specialists have developed and patented a technology for contactless control of industrial robot manipulators using hand gestures of the operator. You no longer need remotes, keyboards, joysticks, or special gloves with sensors to perform precise operations, such as grabbing, moving, or assembling parts. It is enough for the operator to show the desired hand movement in the air, and the robot repeats it in real time, accurately reproducing the position of the hand and fingers.

The key technical element of the system was the depth camera. Unlike the usual one, which captures only color and brightness, it measures the distance to each object in the frame and creates a three-dimensional model of the operator's hands. This allows the developed system to recognize not only the movement of the entire hand, but also the smallest changes in the position of the joints and fingertips in space, the developers told Izvestia.

First, the robot is trained with a set of basic gestures that are used as control commands, for example, to switch to training mode or start autonomous operation. After that, the operator can demonstrate the complete sequence of actions for performing a complex operation by giving commands with hand movements in the air. The robot repeats the gestures in real time, memorizing the trajectory, speed and orientation of the manipulator grip. In the future, he is able to reproduce the operation independently, and if necessary, the operator can remotely adjust his actions using the same gestures.

According to the developers, the very first practical application of the technology took place back in 2019 as part of an educational project to create a manipulator for the TNPA TurtleROV2 underwater robot (a compact robot for working at depths up to 400 m).

— Based on this experience, we have refined and adapted the control algorithms for industrial manipulators. We see the closest and most obvious scope of application in the framework of the concept of unmanned production, which is being developed by Sirius University. Our gesture control technology is a natural and safe way that allows the operator to remotely train, configure and adjust the operation of industrial robots in such industries," said Andrey Shipatov, Associate Professor of Mathematical Robotics at Sirius University of Science and Technology, Associate Professor at the National Research University MIET, CEO of Elmix.

Hit the precision: a robot meter will help create optics for power plants in orbit

What new advanced technologies will allow us to develop surface smoothness testing?

Scope of application of gesture control

The developers noted that gesture control is especially important where full automation is not possible or has not yet achieved the required accuracy. For example, in aggressive production environments, where the presence of a human operator next to the robot is unacceptable, and there is no possibility of reconfiguring or calibrating the equipment installed on the manipulator itself. It is these practical limitations that have led to the creation of a contactless interface that allows remote robot training to perform complex tasks while maintaining operator safety.

The scientists also said that the delay between the operator's gesture and the robot's movement corresponds to the time it takes to receive and process a single image frame from the depth camera, which is about 50 ms. This speed, in their opinion, is quite sufficient for many technological operations that require a high reaction speed.

The technology will be useful for use in industries where it is impossible to pre-program all actions due to their complexity or frequent changes. Plus, it will allow them to be reprogrammed without the participation of a specialist or programmer, Anton Averyanov, CEO of the ST IT group of companies, TechNet NTI market expert, explained to Izvestia.

— Such technologies have been demonstrated by large foreign companies involved in the development of robots and robotics, but this is the first such development presented publicly in Russia. Let's hope that the method will be applied in work as soon as possible, which will help raise the quality of domestic production robots," the expert noted.

In a number of areas, this approach may have a promising continuation, but it should be understood that the development does not involve teaching robotic manipulators to understand gestures, said Anton Pyrkin, Dean of the Faculty of Control Systems and Robotics at ITMO University. There is a set of additional equipment for retrofitting a robotic arm integrated into one control system and generating commands by interpreting external signals from a depth camera.

— A large number of works are known where, instead of a depth camera reading gestures, neural interfaces are included in the control circuit that measure human encephalograms related to the electrical activity of the human brain, allowing robots to be controlled virtually by the power of thought. Here, most likely, a solution with a very high resolution of the camera and its processor is proposed, allowing you to understand not only the gestures themselves, but also their parameters that affect the desired position and orientation of the robot with great accuracy," the specialist said.

The use of this control method is possible where it is necessary to use complex manipulations, but only a person is able to control their execution with specified accuracy characteristics, experts emphasized.