Generating Drone Footage

High-quality data in form of images or videos is the basis for successful object detection training. The better and diverse the quality of this material is, the higher the chances are of detecting real-world defects on the plane. However, since there is little to no real-world material to train the object detection model on, generating training material using 3D software was a promising alternative to address such an issue as this material has high requirements regarding quality and similarity to the real-world scenario. To achieve these goals, our team decided on the use of Unreal Engine 4 (UE4), which gave them top notch picture quality and light approximation using Nvidia’s real-time ray tracing. On this basis they sought out to simulate the drone flight as accurately as possible, to account for camera movement, reflections, and shadows in the later footage. Microsoft AirSim is a software package that integrates into UE4 to create such a simulation, an additional advantage of this software combination being the creation of object segmented images. This allowed us to create labeled images without having to perform this manually, which of course would be required for real-world footage.

Simulating Damage

With UE4 and AirSim as tools, zeroG first built a realistic scene in which an A320 plane was standing in a hangar and then created a drone path on which the pictures were taken. To create variation to this scene they used UE4 C++ API to simulate different artificial lights from overhead lamps as well as natural light from outside the hangar at different times of the day.

The plane was then virtually damaged, whereby we selected very common defect types namely: scratches, dents, and lightning strike burn marks. A combination of real-world data was implemented where images were available, and the defects were simulated procedurally. To generate realistic-looking defects from a simple photograph Materialize was used by extracting normal and height maps. This was necessary for the 3D software to realistically display the defects as actual objects and not as flat images on the planes surface. Where this approach was not feasible, the Python API in Blender was used to generate dent defects including its normal, height, and alpha maps. Due to this a wide variety of defects were readily available. Once the specific positions were mapped in the hangar 3D space we applied these defects on pre-defined areas of the plane again using the C++ API of UE4 to vary position, rotation, and size. As a final, we rendered several hundred images per light/defect position variation using RTX2080Ti GPUs.

Detecting Defects

Since essentially two images are created per timestep, one as a real camera image and a second object segmentation image, we were able to use the OpenCV library to auto-create labels. The labels and the corresponding image then serve as input for a MaskRCNN detection model. If real-world camera footage were to be used this process would be one of the most time-consuming and expensive processes since every defect would have to be highlighted by a human.

We used high-resolution inputs since the defects are comparatively small and it is extremely challenging for the model to detect them. In the next step we aim to transfer the results from our artificial images into the real world by testing the performance of the detection model on real-world images and as soon as real drone footage is available our training material pool can be enriched using these images.