Software and data processing tools

Automatic defect detection is a continuously advancing topic in research that lacks an effective market implementation in the case of shipping industry. Recently, there exists a boost for autonomous inspection of industrial, transport and building infrastructure through the synergy of robotics and computer vision. A first approach on rust detection on vessels has been developed within the MINOAS and INCASS frameworks. 

The automation of corrosion and crack detection via non-contact and non-destructive techniques, instead of electrochemical methods, is an elaborate research problem. Typical methods perform image analysis on RGB data from metal surfaces. The various employed methods are mainly categorized in two groups; the ones based on an automated detection (e.g. on the wavelet domain, thresholding, spectral band combinations and analysis, image segmentation, boundary or shape analysis); and the methods based on image classification procedures. 

ROBINS will promote the state-of-the-art in research and bring productive cutting-edge technologies closer to shipping market via the robotic integration of advanced defect detection and recording. Novel platforms of integrated lightweight sensors will be built, specifically oriented to defect detection. These platforms will take advantage of the modular structure of the proposed robots, aiming at adaptation on each case.

Simple image feature extraction may not be enough to detect damage in ship structure due to complex lighting conditions and great diversity of other conditions within data obtained from different robots. In this scenario, manual definition of features for defect representation is not feasible. Thus, digital image based defect detection should be built on a basis of highly robust machine-learning approach such as convolutional neural networks (CNN), which already showed good results in problem of object classification in photographs and looks promising for industrial inspection applications. In contrast to manually designed image processing solutions, deep CNN automatically generate powerful features by hierarchical learning strategies from massive amounts of training data with a minimum of human interaction or expert process knowledge.

Development of deep learning-based approaches for corrosion detection by means of bounding box regression.

Development of deep learning-based approaches for corrosion detection by means of semantic segmentation.

CAD and image processing software tools play a key role in the RAS-assisted inspection loop and therefore their development is a main objective of ROBINS. The software is expected to:

• Devise a 3D numeric model of the confined space subject to inspection by means of image processing algorithms capable of combining together 2D pictures, and/or by means of meshing algorithms capable to devise textured meshes from point clouds and photographs
• Provide virtual tours of the space subject to inspection. The user should be given the possibility to examine accurately the details of interest by moving in the 3D virtual space and setting the orientation of its viewpoint according to his need, and having always a detailed rendering of the surface observed consistent with the viewpoint
• Provide the possibility to add hotspots and/or associate additional information to selected parts of the 3D model (augmented virtual reality model)
• Identify critical or suspect areas from the analysis of visual data acquired during the inspection and highlight such areas in order to provide a valuable guidance to the surveyor

3D reconstruction of real world objects has been an important research area for decades in computer vision as well as photogrammetric community. Accurate surface reconstruction has been established as a necessity for a variety of mapping, modelling, and monitoring applications (e.g. thickness measurement, cracks detection, coating condition assessment, evaluation of mechanical damages).

The most fundamental step required for surface reconstruction is generation of 3D point cloud. A point cloud is basically a large collection of points that are placed on a three-dimensional coordinate system. Two main methods of extracting point clouds exist: 3D scanning and photogrammetry. The 3D scanning techniques can be further subcategorized into laser scanning and structured light scanning. Spatial data is obtained by moving the laser head or the structured light cameras relative to the object being scanned to directly obtain point clouds of the object (or sections of the object if it is too large). With photogrammetry techniques, 3D point clouds are generated from a large set of images by matching 2D points and edges which are transformed in the 3D data by forward ray intersection. The following steps are required to extract point cloud data from a set of photos:

1. Feature recognition – each photograph is analyzed and key features are identified, which are invariant to scale and rotation and may be potentially used to align the photos.
2. Feature matching – features have to be matched between photos.
3. Alignment of cameras – the coordinates of the cameras relative to each other and the recognized features is calculated by minimizing the error between distances on images and expected distances for all cameras. Minimization is usually performed using Levenberg-Marqardt algorithm and is collectively known as bundle adjustment.
4. Construction of dense point cloud – once the cameras are aligned and distances between key features are known, construction of the dense point cloud can begin. This step is the most computationally intensive and can be performed using forward ray tracing.

While point clouds can be directly rendered and inspected, usually point clouds themselves are not directly usable in most 3D applications, and therefore converted to polygon mesh or triangle mesh models, NURBS models, or even CAD models using surface reconstruction techniques. Different approaches to produce point clouds (e.g. laser scanning or photogrammetry) lead to variety of properties and imperfections that pose significant challenges for surface reconstruction methods. These properties distinguish the class of reconstruction methods existing today which range from methods assuming a well-sampled point cloud, generalize to arbitrary shapes, and produce a watertight surface mesh, to methods that make very loose assumptions on the quality of the point cloud, operate on specific classes of shapes, and output a non-mesh based shape representation. The methods available are based on explicit meshing (e.g. ball pivoting algorithm, Delaunay triangulation) and implicit reconstruction (such as Moving Least Squares (MLS), Radial Basis Functions (RBF), Poisson surface reconstruction). It should be noted that the scale of acquired data is quickly growing: software tools no longer deal with individual shapes, but with entire scenes, possibly at the scale of entire cities with many objects defined as structured shapes.

The UIB team has been developing several methods for the detection of coating breakdown and corrosion (CBC) in images taken from vessels during inspection operations. The methods that have been developed are based on Deep Convolutional Neural Networks (DCNN), either for object detection or for image semantic segmentation. More precisely, three different approaches have been considered and developed:

1. Regression of oriented bounding boxes parameters for defect detection. This is a two-stage detector trained for detecting bounding boxes containing CBC. The first stage is a modified version of the Single-Shot Multibox Detector (SSD), where an image pyramid-based approach is adopted to contemplate the detection at different scales and so capture minor details if needed. The output of this stage is a collection of bounding boxes containing the defects sought. The second stage implements a lightweight CNN for regressing the parameters of the oriented bounding boxes better fitting the defects that lie inside the unoriented bounding boxes produced by the first stage. Unoriented and oriented bounding boxes resulting from the detector for some images can be found next.
2. Fully supervised semantic segmentation. In this case, we have adopted a Fully Convolutional Neural network that has been trained for the CBC detection case using up to four different loss functions to make the network capable of small defects detection: the Focal Loss, the Dice Loss, and the standard Softmax and binary Cross Entropy losses. Examples of results for this approach, after training using the Dice Loss, as the winning loss function, can be found next.
3. Weakly supervised semantic segmentation. With the aim of reducing the cost of image annotations for training, in this point we propose a weakly-supervised segmentation approach based on U-Net, in which, similarly to Attention U-Net (AUN), we embed Attention Gates specifically modified for this application. We make use of coloured scribbles for the different classes as weak annotations. By means of a superpixel segmentation stage we generate a training pseudo-mask on the basis of the intersection between the scribbles and the superpixels. Both the weak annotations and the pseudo-masks are used by the loss function during training. The latter has been specifically devised to counteract the possibly incorrect labelling of certain pixels due to the vagueness of weak annotations and the pseudo-masks generation process. To this end, the loss function comprises three terms, namely a partial cross-entropy term, a novel Centroid Loss term, and a regularization term based on the mean squared error. They all are jointly optimized using an end-to-end learning model. Some detection results for this approach can be found next.