The crawler, developed by GE Inspection, is intended to be used for thickness measurements and other close-up investigations.
Project Achievements - Phase 1
The main findings from the field trials in a real cargo hold on-board a bulk carrier were that the position of the probe head was not actuated with enough degrees of freedom in order to reach all the points of interest on the stiffener structures.
The next phase of design therefore focused on improving the agility of the probe holder and to improve the thickness reading capability.
Besides a few other improvements, an interesting modification, which was explored by the next design update, was the position of the LIDAR sensor. It was to be transferred to the moving actuator.
The existing platform was to be extended while maintaining the capabilities of the original system. The chosen design was a second layer of waterproof housing on top of the existing shell. This approach allowed for seamless and protected integration of an on-board computer, a ultrasound testing device (UT) depth cameras, safety sensors and a scanning LIDAR.
Figure 1: Platform Design with additional features
Global Localization
Global localization of the robot within the environment of a cargo hold is required in order to report any measurements in a structured form. The location of the measurement needs to be reported with respect to some agreed coordinate system. Additionally the state of the robot (robot position, sensor position, actuator state and off course the thickness reading should be recorded and reported in a consistent way.
In order to achieve the global localization, a system was developed which allows for deriving the robot position from a sensor fusion of a scanning LIDAR sensor, IMU data and wheel odometry. The system makes use of a Monte Carlo filter, also known as a Particle Filter in a combination with a Kalman Filter in order to obtain the most likely position for a robot in a given environment model with current and past sensor input. For this to work, the environment should be modeled with 3D surfaces prior to the robotic deployment.
Figure 2: Global Localization of the Robot within a 3d surface model of a ballast-tank. The red lines represent the evaluated scan lines of the LIDAR sensor.
Figure 3: Robot Localization inside a Cargo Hold of a Bulk Carrier. The dots represent locations where thickness readings were performed.
Local Environment Perception
In order to approve operator feedback and to allow for semi-autonomous and autonomous behaviors, it was essential to implement improved three-dimensional perception capabilities on the robotic platform.
The installed array of three depth cameras (one to observe the surface on which the robot is driving, directly in the front and two for looking forward and to the sides), was first tested on a prototype inside a ballasts tank.
Figure 4: Point Cloud from front mounted depth cameras as presented to the operator during the first tests.
After bringing together the point cloud from the depth cameras with the global localization system, the operator is now presented with an overlay of three-dimensional spatial awareness. The global localization system provides feedback about the position, orientation and state of the robot. The depth cameras aggregate their information into a pointcloud overlay to confirm that the maneuver space is indeed free of obstacles and that the dimensions of the environment model actually match the reality.
Figure 5.a: View of the same robot from onboard camera
Figure 5.b: View of the same robot from global localization system
Figure 5.c: View of the same robot from with the Depth Camera pointcloud overlay
First field tests showed that the position of the UT sensor between the front-wheels of the robot, though perfect for sensor protection and reliable measurements, was not optimal for the use-case of ship inspection. It was decided to add an actuator to the system to allow for more sensor positioning flexibility. The difficulty with the probe mounted on an actuator arm was that a new probe-holder had to be developed and designed.
Figure 6: BIKE platform with ROBINS extensions and UT probe-holder on an actuator
Crucial for a commercially attractive robotic inspection of ships is the ability to reach remote, not directly accessible location, be it due to height or in confined spaces. On a technical level this requires a crawler that is able to handle at least 90° corners and edges (to traverse from one plate/all section to the next), preferably also 180° edges (for crossing stiffeners). A promising approach is the wheel-parallel-to-wheel approach, which is already working under lab conditions. In this project, the range (transitions angle, wall orientation, size) as well as the robustness against aggressive environments will be improved. This project shall adapt such mechanisms and integrate them into the BIKE robot.
For efficient operation and user acceptance the simplicity of handling and navigation of the inspection tool cannot be underestimated. ROBINS shall develop control strategies based on multiple sensor inputs to support the operator in reaching his/her inspection target quickly. The operator shall be able to concentrate fully on the very inspection task and must not be deviated by complicated steering a complex robot. We will address this gap using multiple sensors and appropriate control strategies. This can involve: odometry (1), inclination sensors and gyros (2), cameras (3) edge detection (4), distance sensors (5) and information from outside such as from the flying platform (6) or a fixed base station (7).
For remote UT inspection, the UT probe has to be brought to the inspection location by a manipulator and then be put in close contact w/ the test object. This is solved for access path free of obstacles by sliding the probe on the surface or simple lifters from the launching point to the inspection position. However, such standard probe systems prevent passing obstacles – such as 90° corners – along the access path. Additionally, they are not flexible enough to position the probe at some critical locations, such as welds. In ROBINS, an articulated probe handling tool shall be developed and integrated to both position the probes flexibly as well as to move it out of collision situations.
In remote inspection, the probe is far away from the operator controlling the data acquisition. The data quality (electrical noise) and the need to feed couplant liquid to the probe contact limits this distance or impairs signal quality. We shall develop or adapt and then integrate a miniaturized UT control system and couplant feed system has to be integrated on-board the robot.
Robotic crawlers for inspection of vertical or even overhanging structures are both an active area of Research as well as a field of intense industrial development. As the typical structure in ships are made of (magnetic) carbon steel, magnetic adhesion is typically used for generating the positive contact force required to provide both adhesion and traction against gravity.
In this project, other adhesion principles such as vacuum or electrostatic adhesion are not considered. Magnetic crawlers typically run on magnetic wheels or tracks, providing both adhesion and propulsion. These wheels can easily deliver magnetic forces 2 to 5 times the weight of the robot, thus providing a reasonable
safety margin against detachment and finally falling, even under sub-optimum surface conditions, such as paint and dirt layers or rust. Problematic is the quick contamination of the wheels/magnets w/ magnetic particle. Thus, the surfaces have to be sufficiently clean in order to guaranteed sufficient inspection time between cleaning procedures.
The main problem of crawlers based on magnetic wheels is the limited ability to pass non-flat obstacles. As soon as one (or more) wheel touch two surfaces, e.g. in 90° corners, tremendous forces are needed to free the wheel from the first surface and then move on.
Often the motor forces and/or the friction at the contact surface are not sufficient to escape this locked-in situation. Several approaches have been proposed to overcome this limit: multiple wheel configurations, moveable field magnets instead of magnetic wheels, active lifters, passive lifters.
For ship inspections, several crawlers have been developed to clean and inspect ship hulls. However, due to size and limited obstacle handing ability they are not suited to inspect other parts of the ship, such as structured cargo holds, stiffeners or (ballast) tanks. Additionally, they lack robustness and stability during handling such obstacles.
Due to the large magnetic forces (up to several kN), placing the robots on a magnetic structure and even more removing it can become a difficult and dangerous task. Dedicated placement tools or adjustable magnet positions are used to enable the operator user friendliness and safe handling.
It is industrial practice to manually control the robotic crawlers along the surface, thereby requiring fulltime, uninterrupted attention of a well-trained operator. Limited support for dedicated scenarios support comes from builtin sensors: inclinometers enable straight vertical motions and line-following sensors control the robot along a given feature, such as an edge, a weld or an artificially placed guidance structure.
Project Achievements - Phase 2
An important and difficult maneuver when using the BIKE platform is the transition from one plane to another by means of crossing through a concave corner or by crossing over an edge (convex corner).
Figure 8: BIKE crossing over a convex corner (edge)
Figure 9: BIKE crossing over a concave corner
This maneuver relies on precise alignment of the BIKE robot with the edge. Furthermore, during the crossing, the operator needs to interpret the visual feedback correctly and manually adjust the robot controls for successful completion of the task.
For execution of the corner-crossing maneuver itself, the BIKE controller needs to switch from front-wheel to rear-wheel control, depending on the position of the robot in the corner and the direction of the maneuver. So far, this switch of wheel control schemes was either not performed at all (resulting in wheel slippage), or the operator needed to judge when to switch based on the visual feedback.
With the current implementation of the local controller, also an improved corner crossing controller was implemented, taking advantage of the IMU and Odometry based robot pose estimator used for path following and map building. Using only gyro-data from the IMU and wheel rotations, this pose estimator generates very accurate (local) robot pose updates. For the corner crossing, mainly the gyro-data is used to determine in what state of the transition from one plate to the next the robot currently is.
Enviromental Model Generation
The global localization is based on an accurate model of the static environment in which the crawler robot is operating.
Obtaining the environment model is the first step for any deployment of the crawler.
Global Model Generation by High-Resolution 3D Scanning LIDAR
In the case of the cargo hold encountered in the field trials a high-resolution 3D LIDAR scan was used to obtain the model. A total of seven scanning locations were used and the point-cloud data was then manually registered for data reduction and meshing.
Figure 10: 3D LIDAR pointcloud from a single scanner location
Figure 11: 3D LIDAR based mesh as obtained from manual registration, clean-up and meshing of seven point-clouds. This model is composed of 400’000 triangle faces
Local Model Generation by 2D LIDAR on Actuator Arm
After the field trials it was decided to create an experimental setup with the LIDAR sensor mounted on the movable actuator, which was originally only carrying the thickness measurement sensor.
The option for tilting the LIDAR measurements plane by more than 90° removed the necessity to re-orient the robot periodically when going straight in large environments. Another benefit of this movable 2D LIDAR was that it basically converted the BIKE crawler into a crude 3D LIDAR scanner, which was explored and is described in the next section
Figure 12: Concept for LIDAR on actuator arm
With the updated LIDAR configuration, it is theoretically possible to generate a point-cloud, and consecutively a mesh, from a series of 2D LIDAR scans taken at different known tilt angles of the actuator arm. If the robot is localized by a global localization system, based on a model of the environment, it is even possible to collect several such point-clouds with the 2D lidar and register them into a single cloud for meshing and creating a local model of the environment.
Such local modeling capability would greatly improve the robotic performance in areas, where the global scan was unable to reach due to shadowing. It could also allow for inspection without a global model entirely. The robot could basically generate the model on the go.
To evaluate the feasibility of this approach, a test setup in the lab was used. The crawler robot was placed in a cylindrical mockup and 2D scan data was recorded when the robot was motionless, and the actuator arm was moved very slowly.
While it would be possible to perform the manual point-cloud registration and meshing process, which was used to generate the global model of the cargo hold, it would not be very practical to use this process during an inspection mission. Instead, an automated process was implemented based on using an open source library for point-cloud registration, filtering, and meshing.
The implemented process is:
- Register all point-clouds into a single cloud
- Decimation (reduce the number of points for computational reasons)
- Meshing
- Clean-Up
- Create High-Res and Low-Res Mesh for localization
The result is a closed mesh, with consistent face normal and shows only a slightly noisy surface. The method could be used to generate a partial model of a confined space in more or less real-time and as such could allow for skipping the expensive modeling step entirely in the future.
Figure 13: Point-clouds as recorded from the moving 2D LIDAR on the actuator arm in the lab test
Figure 14: Meshed section of the Lab Mockup
The success of the implemented registration and meshing tool motivated a re-run on the data obtained from the field-trials in the actual cargo hold. The automated point-cloud registration created less error in the final point-cloud and the more sophisticated meshing method was able to output a high-resolution mesh for the operator as well as lower resolution mesh for the localization algorithm.
Figure 15: Result from automated point-cloud registration and meshing: High-Resolution mesh for visualization to the operator.
Figure 16: Result from automated point-cloud registration and meshing: Lower-Resolution mesh to be used by the global localization algorithm.
Due to the limitations for going on-board actual ships or even for travelling abroad for tests in the testing facility due to the COVID-19 pandemic, the project partners were looking for alternative options. One option was to train a robot operator who is local and ship the system to the testing facility. This was regarded impractical for the BIKE robot, due to the complex nature of the crawler and due to the need for training on-site, which was also prohibited by the pandemic. Instead, it was decided to modify the robotic inspection system, to be operated remotely with only some assistance from the local staff.
The remote-control setup allows the robot operator to access the control PC, power the water pump through a web-interface and to re-boot both the robot control unit as well as the control PC remotely if necessary.
Several modifications to the method of controlling the robot were implemented:
- A maximum driving distance is provided to the robot prior to any motion command. This should prevent the robot from moving away in the case of a connection interruption to the remote operator.
- The operator uses a software joystick to replace the physical joystick usually connected to the USB port of the local computer.
- Only one computer is used to control the robot – hence all software is run on the same PC and shares a single screen, which is shared remotely. – This has implications on the computational load that can be put on the computer and requires the operator to change from the robot control window to the measurement window. For moving the robot while the thickness readings are shown, the remote controls can be placed over the measurement window.
- The robot top speed was reduced to allow for reasonable operator-in-the-loop control even over a high latency connection.
For safe operation, an accurate model of the test mockup was generated and validated prior to the remote deployments.
The local staff would need to place the robot on the test specimen and also retrieve it after the test. Additionally, they would need to provide a bucket of water and stay close-by to observe the activities and ensure safe operations.
The interaction with an instructor or surveyor and the robot operator was set up in the form of a shared online meeting with access to an overview camera in the testing facility. – hence the surveyor can give instructions and get a feedback on the execution.
Figure 17: Robot placed on mockup in testing facility for remote testing operation
Figure 18: 3D view of the remote testing environment presented to the operator
Figure 19: View to the operator when lowering the thickness reading probe in a remote test
Figure 20: View of thickness reading A-scan to the operator once the pump was turned on to provide water for coupling the sound into the metal and back to the sensor