Scientists work on 'superhuman' vision systems for robots

If you want to find out whether your robot can see through smoke, well, you’re going to need some smoke.

But a University of Pennsylvania student got a shock when they began setting up a late night experiment to test such a robot.

Shortly after flicking the switch on the smoke machine, a loud fire alarm went off.

“The whole building got triggered,” says Mingmin Zhao of the University of Pennsylvania, smiling. “My student called me. He was very surprised.”

The incident was a minor setback for the team developing a robot equipped with a innovative radio-based sensing system.

Radio waves could allow robots or autonomous vehicles to see through thick smoke, intense rain – or even around corners. Such waves can even detect concealed weapons.

But simulating visual imagery based on radio waves is an unusual approach for robots and autonomous vehicles. Much more established in those fields are regular optical cameras, light detection and ranging (Lidar), and other sensors.

However, Prof Zhao and his students have developed a potentially powerful way for robots to see using radio waves.

Of course radar, which uses radio waves, has been used for decades to track aircraft, ships and the weather.

But the spinning array on Prof Zhao’s robot throws radio waves in all directions.

An on-board artificial intelligence (AI) system then builds a 3D view of the environment with this information.

“What we have been trying to do here is basically help robots obtain superhuman vision – to see in scenarios where human eyes or traditional visual sensors cannot,” explains Prof Zhao.

He suggests the technology could help a future search-and-rescue robot save people from a burning building.

Subsequent tests of the bot used a clear plastic box full of smoke placed around the its spinning equipment, in order to avoid triggering any nearby fire alarms.

Although humans can’t see them, radio waves are a form of light in the sense that they are part of the electromagnetic spectrum, which also includes X-rays and gamma rays. Only a small part the spectrum is classed as visible light.

Being light, radio waves can reflect off surfaces and materials, though in a slightly different way to visible light. Prof Zhao and his colleagues have designed their robot so that it can sense these radio wave reflections.

The crucial factor here is that radio waves are much longer than visible light waves meaning they are not blocked by tiny smoke particles.

Prof Zhao says he has also been working on adapting the technology so that the robot can see part of the way round a corner. Think of it like a hall of mirrors, he suggests, just for radio waves rather than visible light.

“It’s really very interesting work and quite impressive,” says Friedemann Reinhard at the University of Rostock in Germany, who was not involved in the work. In 2017, Prof Reinhard and colleagues described how Wi-Fi signals could allow spies to see into private rooms.

One slight limitation is that the spinning array cannot, by definition, see in all directions at once. Prof Reinhard says that a lot of data processing carried out by the system appears necessary to clean up the image that results from this spinning device.

However, the robot sends out radio waves in the millimetre wave band (waves that are between one and ten millimetres long). This is the same technology used for some 5G installations.

“That’s potentially very attractive, it’s a very well understood, cheap technology,” says Prof Reinhard. “I certainly would love to see a self-driving car driving only on radar.”

But it is possible to avoid using spinning radio-emitting devices to get a full picture, says Fabio da Silva, founder and chief executive of US firm Wavsens, which is also developing radio-based sensing technology.

“We created an algorithm that allows you to sense the entire space instantaneously and continuously so we don’t have to spin our antennas,” he says.

He describes the system as akin to echolocation, used by bats. It sends out radio waves and “listens” to how the waves come back, which reveals the shape of whatever they have hit.

Some researchers have used radio waves to detect concealed weapons such as hidden handguns and knives.

Radio waves can even “fingerprint” the details of a room. Then, if scanned again later, this would reveal whether any objects in the room have been moved.

Last year, scientists in Germany proposed that countries use this method to scrutinise other countries’ management of their nuclear weapons stockpiles. It would be one way of knowing whether someone had been moving the warheads around, for example.

Separately, Luana Olivieri at Loughborough University has explored using a different form of non-visible electromagnetic radiation, terahertz waves. These are shorter than radio waves but longer than visible light waves. “This wavelength is particularly unexplored,” says Dr Olivieri.

It is possible to see through objects and analyse materials using this form of radiation, she adds. Such a system could even, in theory, identify specific drugs by detecting their chemical structure.

But while seeing through materials could help a rescue bot find a trapped person in some future disaster, it also has other applications. Police forces and militaries have access to radio-based technology that allows them to see through doors and walls, to some extent.

“Warfighting is definitely one market that it caters to. It can be used to find and kill someone,” says Mr da Silva. He has exhibited Wavsens’ technology to the US Department of Defense and the Israeli Ministry of Defense, he says.

And yet these applications are not entirely surprising, suggests Prof Reinhard, who points out that a range of emerging technologies have, in principle, made it easier to detect and attack people.

“Maybe radar sounds scary – but drones and cheap cameras are the much more dangerous thing,” he says.