When Software Meets Hardware, or: How AI Can Protect Power Lines

Better, faster and, above all, increasingly automated – this is the vision for many sectors of industry, from the automotive sector to energy suppliers. One solution that is becoming more and more popular is artificial intelligence. We spoke to Bernd Schröder, software developer in the Physical Design Tools & Software group at Fraunhofer IZM, about the different applications of neural networks, how they learn, and what it is like to swim against the tide as a software professional at a hardware institute.

Fraunhofer IZM, Fraunhofer HHI, and 50Hertz are working on the inspection and maintenance systems of the future. Can you tell us more about this?

Bernd Schröder: In our research, we are contributing to optimizing the maintenance of electricity pylons and conductor cables. The issue here is that there is a risk of rust or structural damage to the cables, potentially leading to their collapse. Normally, these power poles are regularly inspected manually, that is, by trained staff driving or flying along the entire route. This is time consuming and expensive for the companies.

This is where our project CoDro, as a proof-of-concept, comes into the picture: We wanted to show that appropriately detailed photographs and an AI could help detect rust spots and alert the grid operators

This means that no one has to physically drive out into the field; instead, the entire process can take place from the comfort of your desk, working only with digital images.

What exactly did you do?

Bernd Schröder: Our project team (Fraunhofer HHI, Fraunhofer IZM, and 50Hertz) developed the software and the AI algorithm. That was not as trivial as it may seem, since the algorithm has to be able to precisely distinguish between, for example, rust and dirt.

The job of Fraunhofer IZM on the project was to develop the underlying system concept and to set up a database where the photos are uploaded and categorized. This was a key point, because the AI first has to learn what it is supposed to learn. To do so, a person must sit down and annotate the relevant places on the photos, where there is rust, for example. Then our partners at HHI used this information to teach the algorithm.

At first, this project had nothing to do with hardware or drones. But at a later stage, drones may be used to automate the process. Our vision is a pre-programmed path that drones could fly along independently. For now, that is not permitted by law in Germany, but it is still a vision for the future.

Image databases, labelling, annotation tools, and web interfaces – What do all these terms mean in practice? 

Bernd Schröder: You have to imagine it like this: There are different routes in these power grids and, at the end, there is a computer or a server where all the data accumulates. On this server, all the data, i.e. the images and the image analyses, is managed in a database. We have to feed this database with meta-information, like which actual mast belongs to which image, at which height, and so on.

In order to get such meta-information in, we have built a web interface for the server. The customer can enter all his data in a special form, press a button, and have his data analysed by the AI algorithm.

And labelling and annotation is exactly what we do up front, e.g.  manually annotating images with metadata. As I said, that is for training the AI.

What exactly do you mean by artificial intelligence in the project CoDro? What did it look like in concrete terms?

Bernd Schröder: In this case, it was done with a neural network. More precisely, it looks like this: You have an image, a matrix of pixels, and each pixel has a colour value. All this pixel information is simplified down to form certain patterns. The whole thing is then processed with the help of mathematical formulas and curves. Next, the so-called weight vectors come into it, which determine how strongly a certain feature is responsible for the AI making a decision.

In the end, we get a type of black box calculating the entire network. At the beginning, the end result will be completely random. It is only because an actual person determines which results are expected that the algorithm can compare its calculation with these expectations and the results at the end. That is the whole problem with neural networks and artificial intelligence – figuring out what the structure of the network should be, how complicated it can get, and what information to provide for learning.

There are many challenging moments. For example, when we have too little information, the AI learns those images very quickly. This means we will have images later that it will not understand. Or vice versa: if we have too much data, the algorithm does not work well either. So you need just the right quality and quantity of learning data.

In your opinion, what value is there to be had from AI algorithms?

Bernd Schröder: The value is always felt by the person who uses it. So it is quite obvious for the customer. They want to automate certain processes, for example, but the problems cannot really be described mathematically in the relevant spots. This is where AI comes in.

For us, AI can deliver results more quickly, because the faster you have your solution, the faster you can go to market. That is often the motivating factor in design: saving time, saving effort, and also eliminating sources of error.

Can other failure mechanisms also be analysed with AI?

Bernd Schröder: In this case, we focused exclusively on corrosion. It is still a proof of concept, so we first wanted to show that our project is feasible at all. And, of course, that our predictions are of good quality – also in general use, i.e. when the sun is shining, or when it is stormy, or even when the photos have the wrong angle. In any case, our results have already been evaluated, and the feedback is very positive.

Speaking generally, the AI algorithm we developed could be applied to other applications, but the problem that this neural network solves is, of course, very specific. If you have a different problem, then you have to completely redesign it and start over with the learning process. But it might be an advantage to have an already pre-trained network.

What were the biggest challenges during development?

Bernd Schröder: The biggest challenge was definitely to build the system in such a way that it would work without problems for 50hertz’s customers. They have very strict security criteria for their IT infrastructure. We had to develop everything here on our premises – locally and, ideally, so that we could give them a USB stick that they can take and plug into an available computer and test the entire system as if it corresponded with their network infrastructure. 

We have already found a good solution for this, so that we can skip the installation efforts.

Will you stay working on power lines, or are there other topics that get you excited?

Bernd Schröder: For the ASTROSE project here at Fraunhofer IZM, we still have a lot of work to do on power lines.

Another project I’m looking forward to is all about developing an AI for the automotive industry that automates or improves the layout and system design of all the electronics, so that fewer redesign cycles are needed for components. One idea is to analyse known problems in order to predict whether a new design is good or bad. And we will specifically be responsible for PCBs. So it is about designing and analysing PCBs that will be used in automotive applications.

What is it like for a computer scientist to work in a hardware institute?

Bernd Schröder: You are, of course, absolutely right: We are a hardware institute through and through. As a computer scientist, I belong to what you could call a small minority. But our group actually works exclusively on software. That’s why we are always cooperating with other working groups or departments, because we can’t do it alone – without hardware, our developments don’t make any sense at all.

For example, we are currently running a project on wire tracks. The point is that there are always different electrical voltages, different electrical potentials on the different copper areas of a circuit board. And depending on the requirements and on the system, these need to keep a certain distance from each other. If there is only one layer, you can still calculate it manually, but if you have a modern PCB with multiple layers and perhaps multiple boards in your system around the PCBs – then it’s not trivial at all. It is exactly for this purpose that we developed an algorithm to calculate these distances for different surfaces, compare them with the required minimum spacing, and get the right graphical feedback.

This is an example that shows why we always work together with other hardware groups: To get the best results.