This is extended project from Autonomous Fire Figihgting Robot Arm to improve fire sensing and localization. The previous project limited fire sensing to 2D imaging, but this project extends to localizing the fire in 3D location relative to the robot and the world frame.

You can check out the source code from the Github Repository

DEMO VIDEO

PROJECT BACKGROUND

Autonomous Fire Figihgting Robot Arm Project’s thermal image processing was able to sense the heatsource and align it with robot arm, but it did not have any information where the heatsoruce is relative to the robot. Since Adroit plans its movement using Moveit, localizing the fire is crucial to the performance of an autonomous fire fighting robot. I added Depth sensing camera and combined a depth image and a thermal image to get depth from the heatsource and eventually localize its location using transformation.

HARDWARE

FLIR Lepton 2.5 - Thermal Imaging Module

PureThermal 2 - FLIR Lepton Smart I/O Board

Intel RealSense Depth Camera D435

3D Printed Cameras Mount

Contour Detection From Thermal Image

The previous method of detecting the heat source was getting the pixel of highest value from the radiometric image. This method was rather unstable due to often bounce of target pixel. Therefore, I calculated the contour from the radiometic image and its centroid to get a target pixel of the heat source.

(contour of a electric heater)

(contour of fire)

Combined Vision Processing

Getting the pixel coordinate (u,v) from the centroid of thermal image’s contour, it’s fed into the realsense depth camera to get depth data at (u,v).

The evaluate depth sensing, the result was compared to measurement with a ruler. The error range was about +-100 mm at maximum when measuring distance from 1.2 m to 0.3 m. The error got greater as the depth camera was more far away from the target object.

When it comes to combining two cameras, thermal image and depth image are deviated from each other. Therefore it’s important align two cameras properly. ropstopic combined_image2 provides combined images of color image and thermal image to provide how well two cameras are aligned while combined_image provides combined images of aligned color to depth image and thermal image. Thermal camera and realsense camera should be calibrated to avoid error due to cameras deviation.

Testing

Testing with Electric Heater

(This moudle can be mounted on Adroit robot arm)

Testing with Fire

(Thermal Image + Color Image)

(Thermal Image + Aligned Color to Depth Image)

Testing with Human

Another application of this project is that we can use it keep track human too.