A bionic sensor skin for semi-autonomous service robots in challenging land and underwater applications

A 50,000 square meter artificial reef off the Baltic coast and a green space following the trajectory of the Berlin Wall: Two places, about 240 km apart from each other and as different as could be. However, only a few months ago, both shared the same purpose as test sites for semi-autonomous service robots equipped with a flexible bionic sensor skin.

It’s an age-old dream: service robots that make life easier and take over the jobs that are too dangerous for us humans. Robots have indeed been used in the automotive industry since the 1960s, but typically for routine work like welding, milling, stamping, or coating. Detecting munitions lost at sea is a completely different matter. Until now, this complicated and risky job is left to special submarines or trained professional divers.

With the participation of Fraunhofer IZM, a German research consortium has been working on two concrete applications of bionic service robotics from 2019 to 2022. Funded by the German Ministry of Education and Research and run by the VDI/VDE-IT association as part of the SensoRob initiative, the Bionic RoboSkin project envisions robotic manta rays that assist in the surveying of underwater structures, like the inspection of underwater pipelines, and similar robotic badgers that could take over geo-exploration in difficult-to-reach places, e.g. monitoring tunnel construction. For robots to work in such harsh environmental conditions, they need modular sensor skins.

RealIZM has met up with Karl-Friedrich Becker, leader of the Assembly & Encapsulation group, and David Schütze, research associate, to get an insight into the 3-year research project. The two IZM employees tell the RealIZM team about the further development of the Robo-Ray and the Robo-Badger, the course of the project, and what applied research means in concrete terms.

Electronics that get under the skin

“A total of six partners were involved in the Bionic RoboSkin research project. Our shared goal was developing a robot platform that is capable of moving autonomously in its environment, using a flexible bionic sensor skin.” Karl-Friedrich Becker explains the thinking behind the project. He is keen to emphasize that the project partners did not start from scratch, but were able to build on previous experience and existing templates.

The central starting point was an existing bionic robot inspired by the manta ray, made by EvoLogics GmbH, a technology company specializing in maritime technology. It already had a camera and a sensor module, but its main fins were previously used only for locomotion and control. It also lacked a human-machine interface (HMI) on board that could support special divers in their missions, e.g. preparing for the expansion of an offshore wind farm. As part of the research project, the fins were equipped with such an HMI as well as complex sensor modules to help control the underwater robot better.

In addition, there were first prototypes for an upgraded underground robot – special sleds equipped with radar antennas – which were levelled up with the ability to detect metal structures, such as munitions, below the surface.

Sensorik Bayern GmbH came up with the concept and design for the highly integrated sensor module, consisting of a pressure sensor and a 9-axis sensor as well as a chip for processing induction signals when the robots move over metallic objects and structures.

The think tank for high-tech textiles, TITV Greiz, contributed a large-size, flexible textile sensor skin suitable for use under extreme conditions (salt water). The fabric serves as a carrier for the sensor elements and provides moisture-resistant electrical connections for power supply and communication.

GEO-DV GmbH, an engineering company active in data management and surveying, was responsible for specifying the function set for the Robo-Badger, contributed know-how from various preliminary geological-topographical projects, and provided a test site for the ground robot.

Embedding the sensor modules and developing the sensor housing

The job of the IZM scientists was system integration: miniaturizing and optimizing the sensor technology. For this purpose, they developed a concept for a modular packaging platform for use in a harsh marine environment. Their focus was on miniaturizing the sensor module by embedding it in a printed circuit board and developing the outer housing for the sensor technology. “What we did was to integrate highly robust and miniaturized sensor modules into the textile skin. It gives the robot touch and proximity detection capabilities and helps it navigate its submarine environment,” says Karl-Friedrich Becker, summarizing the contribution of the IZM team.

 “We decided on a modular, two-part design. The first part of our work focused on miniaturizing the sensor modules as far as possible. We managed that by embedding the SMD components in a printed circuit board,” explains David Schütze. Fitting on a system sized about 23 x 10,5 x 1,6 mm³, the microcontroller, driver, and 9-axis sensor are completely encapsulated in a fiberglass-epoxy composite.

The next challenge was to distribute the modules on the textile skin and connect them into it to form a sensor system. In order to be able to connect the sensor modules to the textile CAN bus system and to protect the structure from water, a waterproof and robust outer housing was required for the sensor system and its connection to the CAN bus. On the side of the CAN bus, there is a circuit board with contact surfaces, each connecting to a sensor module with electrical spring contacts. Mechanically, both parts are held together and protected by the housing. “Thanks to the modularity of the overall system, it was possible for us to reconfigure our sensor module at any time, if necessary, without much effort. It’s quite common to have to make adjustments or repairs during development,” Schütze explains.

Sensor nodes consists of three parts: exchangeable sensor module, connector board and housing filled with fat. Images: Fraunhofer IZM

Robo-Ray: An extremely agile, bionic underwater robot

“In the belly of the manta ray, we have the central control electronics and a position sensor that acts as a reference point for calculating the status quo. A camera and various sensors for recording environmental parameters are also part of the regular equipment.” Schütze describes the Robo-Ray. “One new addition is that the manta ray’s skeleton has been covered with a textile high-tech skin. On its surface, we now have robust and highly integrated sensor modules.”

The Robo-Ray’s fins are each equipped with nine sensor modules. These sensors detect ambient pressure and temperature and module temperature. They also each have a position sensor that measures acceleration, torque, and the magnetic field in all directions. With this data, the sensors’ position can be calculated relative to the central position sensor, turning into an exact measurement of the wing movement underwater.

This data can be used both to optimize the control of the ray and draw conclusions about the actual flow conditions. “By comparing how the manta ray should move and how it actually moves underwater, we can optimize its movement step by step and make it more efficient. We can adapt its movement pattern to the local current conditions. If the underwater robot is supposed to swim to the left, and the current is already pushing it in that direction, we can save some energy to get where we want.”

“The team at Sensorik Bayern GmbH created a fin model. Put simply, it is a flexible board with sensor modules. Using this analogue demonstrator, the Regensburg team developed a digital 3D real-time model of the fin that provides the current fin position and shape,” Becker explains when asked how the IZM scientists knew the ideal positions for attaching the sensor technology. With the help of the digital twin, the sensor modules could be placed in the optimum spot on the fin surfaces.

Touchpads are attached to the top of the fins, which divers can use to communicate directly with the underwater robot (HMI). Underneath the fins, which have a total span of two meters, inductive coils are attached that detect metal structures underwater. Becker is enthusiastic about the system: “Thanks to the flexible sensor skin, numerous application scenarios are possible, such as detecting underwater supply lines or high-explosive ammunition lost at sea e.g. when constructing or expanding an offshore wind farm – and it again helps steer the robot.”

Better together: A swarm of Robo-Rays

Currently, the Robo-Ray can dive down to about 50 meters, where it can record a profile of the seafloor. “The closer the manta ray gets to the seafloor, the better. Visualization is important to avoid risks. This is only possible at moderate depths, due to the increasing darkness underwater. The robot is therefore equipped with a high-power LED to be able to capture visual feedback for sensor events,” says Becker, describing the current swimming capabilities of the Robo-Ray.

In addition, a diver is always present for support and can intervene when needed. In the event that the manta ray needs to be retrieved or redirected, the diver communicates with the robot using the touchpads located on the top of the fins. “In the project, we had very good insights about how the sensor housing must be designed for diving depths of up to 20 meters.”

During the test run in the Nienhagen artificial reef, the team and the divers of BALTIC Taucherei- und Bergungsbetrieb Rostock GmbH successfully demonstrated how the Robo-Ray can dive, swim a distance straight ahead, then turn around and return. With the help of a GPS or distance sensor, the distance traveled can be tracked and incorporated into a digital map – a so-called geographic information system (GIS). Based on this information, the service robot is sent out again to scan the terrain. The next step is to transfer this preliminary work to a swarm.

Videos: Evologics GmbH

“The idea is that a programmed swarm of autonomous Robo-Rays will scan the profile of the seabed,” says Schütze, explaining the future vision. “With the Bionic RoboSkin project, we’ve laid the groundwork for autonomous exploration by service robots cooperating with divers.”

Archaeology without excavators and spades: Ground robot

The Robo-Manta Ray has a companion that also has a spirit animal. The researchers transferred the manta ray’s sensor platform onto a ground robot – a robo-badger. This is also controlled by GPS systems and is equipped with induction sensors and optionally with a ground penetrating radar developed by Sensorik Bayern to detect underground metal structures. It can carry out land surveys such as semi-autonomous geo-exploration in difficult-to-access areas, for example to monitor ground movements in tunnel construction.

“Admittedly, the Robo Badger doesn’t win people’s hearts as easily. It is unfortunately not as cute as the Robo-Ray,” Becker admits about the robot’s lack of a real resemblance to its animal namesake. The Robo-Badger is a wheeled robot that moves around on its six wheels. The rear part consists of a wing-like wooden construction over which a two-meter-wide fabric is stretched. This textile skin of the badger is equipped with inductive coils and a GPS sensor. Data is transmitted by radio to a tablet, which also serves as the control unit. “The limited use of metal in the design of the ground robot is important because we want to use inductive sensor technology to detect metal structures hidden beneath the surface. A metal design would only interfere with the sensor data,” says Becker.

The Robo-Badger successfully provided its proof of concept on several measurement runs on the former death strip in Bernauer Strasse in Berlin-Mitte and on the test site of GEO-DV GmbH. “The badger tells us where there is metal, pipes, foundations, or other structures underneath the investigated area. It provides an insight into the recent history of the site: Was there ever a house here in the 1930s to 1960s? Are there possibly still remains of underground structures? It’s like archaeology without the digging.”

It is not uncommon for a greenfield site to lack structural documentation. The owners often find themselves on terra incognita. In order to be able to plan the costs and development work of a site, test excavations are necessary. The more precise these are, the better., since civil engineers and archaeologists cost time and money.

An exploratory and non-invasive preliminary survey of sites with the help of the Robo-Badger allows initial conclusions about the ground conditions down to a depth of 50 cm and provides clues for test excavations. “In order to penetrate into deeper soil layers, test drives can be carried out with the developed ground penetrating radar,” adds Schütze.

Outlook: Robo-Squids for use in the deep

The RoboSkin research project was completed at the end of 2022. Further work has already begun. For example, a multisensory monitoring system for industrial environmental protection in deep-sea mining is being developed as part of the DeepSea Protection joint project.

Research objectives include pressure-neutral encapsulation, highly robust sealing layers, and the integration of a moisture sensor inside the casing to monitor moisture ingress. The sensor system is expected to enable the acquisition and fine-scale verification of geological and topographic data. It will also provide detailed analyses of resource distribution and an up-to-date view of deep-sea ecology at depths of 3,000 to 6,000 meters. The plan is to set up a demonstrator system consisting of four mobile ground stations (deep-sea squid) and two autonomous bionic submersibles.

Many thanks to Evologics GmbH, Sensorik Bayern GmbH, GEO-DV GmbH and TITV Greiz for providing the pictures and videos.