HDT Adroit 6DOF A24 Pincer Manipulation of a fire extingushier using a thermal camera

This is a Winter Project of MSR Program at Northwestern University

you can check out code at my github repository

The extended project improve fire sensing has been completed! You can check out my Fire & Thermal Object 3D Tracking post!

Demo Video

BACKGROUND & MOTIVATION

The idea of a firefighting robot is not a new concept, and there have been a lot of propositions of using firefighter robots. In fact, a robot called Colossus helped save much of the historic structure by braving conditions deemed too dangerous for human firefighters after Notre Dame Cathedral in Paris caught fire in 2019 [1], and LA Fire Department tested autonomous fire fighting robots in the major Downtown blaze in 2020 [2].

Firefighting Robot “Colossus”

The problems with currently existing firefighting robots are that they are not suitable for domestic or commercial usages because they are big and built specifically to operate in firefighting missions. I believe that manipulating a robot arm to detect a fire and use a fire extinguisher can be beneficial in industrial settings such as warehouses that have already adapted using robot arms. Especially, maintenance costs of fire sprinkles due to frozen pipes can be deducted in warehouses. Additionally, implementing a fire safety feature to domestic service robot arms can appeal to the general crowd.

Therefore, I implemented a firefighting mechanism using HDT Adroit 6 DOF A24 Pincer robot arm (I will refer it as an Adroit through out this post) and FLIR Lepton 2.5 thermal camera for this project. The goal of this project is to detect a heat source (fire), grab the fire extinguisher, and operate it.

HARDWARES

HDT Adroit 6DOF A24 Pincer

Fixed-hose fire extinguisher

FLIR Lepton 2.5 - Thermal Imaging Module

PureThermal 2 - FLIR Lepton Smart I/O Board

SYSTEM OVERVIEW

The Adroit operates on ROS. I used ROS-noetic version for this project.

This block diagram shows the overall flow and communication between different ROS packages for the project. I implemented two packages which are thermal_image_processing and control. Besides, I used packages provided by HDT to control the basic movements of the robot.

CODES COMPONENTS EXPLANATION

Package: thermal_image_processing

This package takes care of detecting a highest spot in the camera view and reading its temperature from the thermal camera. I implemented the package from PureThermal 1 / PureThermal 2 FLIR Lepton Dev Kit (https://github.com/groupgets/purethermal1-uvc-capture).

Thermal Image Processing

Image of a human through a thermal camera

FLIR Lepton 2.5 thermal camera can detect the temperature using its radiometry in a gray scale. The pixel value in the gray scale, for example, shows some value such as 40,000 which represents 400.00 K. The node thermal_detection detects the pixel with the highest temperature and convers into celcious scale. In addtion, it publishes the temperature value and 2D x-y coordinates of the image as rostopic so that the Adroit can execute its movements based on those values.

Package: control

This package controls the Adroit’s arm joints through moveit and pincer through the pincer joint controller. I have implemented the node arm_control for Adroit to detect the fire, grab the fire extiguisher, aims it to the fire, and press the lever.

Fire Detection

arm_control node subscribes to the temperature value and x-y coordinates that thermal_detection is publishing.

Adroit Searching Mode

Joints Movements

The Adroit makes joint movements on its joint 1 and 6 to scan the enviroment until it detects the something with a high temperature value. The threshold for the temperature was set to 70 deg C (~156 deg F) for this project since I used a heater as my heat source, but this value would have been set higher if it were to work with a real fire.

Thermal Camera View

Align Mechanism

When the Adroit detects something above 70 deg C, it will try to align the center of the camera view to the heat source by adjusting joint 1 and 3. The figure above represets its mechanism. When the robot aligns wit the heat source, it remembers joints positions so that Adroit can comeback to this position after grabbing the fire extinguisher. In this case, the Adroit’s joint 1 is a little bit smaller than remebered joint position because the fire extingushier will be located slightly left to the thermal camera.

Grab & Aim

Safey pins and laser pointer

Since I am working with a single arm robot, it is necessary for the picners to be able to two tasks: picking up the fire extinguisher & pressing the lever. Therfore, I implemented custom safey pins to prevent the Adroit to accidently press the lever when it is trying to grab the fire extingshiher. I made two types of the pins that the first one is an electric wire coverd by rubber, and the other one is 16-gauge copper wire. The copper wire one is more sturdy that the Adorit needs to exert more force to press the lever compared to the rubber one. I attached anti-slip stickers on the levers of the fire-extinguisher to get better grip

I used a hard coordinate for the location of the fire extinghisher, and I found this reasonable because fire extinguihsers are usually located at specific and fixed spots. The Adroit sucessfully grabs the fire extinguisher in a rate of 8 out of 10 times, and this error was due to incorrect orientation of the fire extinguisher.

I installed a laser pointer to accurately evaluate the Adroit’s aim to the heat source. When the Adorit goes back to its remembered joint positions, the fire extingusher hose’s aim is aligned to the heat source.

Press

arm_control publishes std_msgs/Float64 to picner_poistion_controller to control the pincers. When the Adroit presses the lever, the fire extinguisher’s aim goes down a little bit. Besides, this project achieved its goal.

FUTURE IMPROVEMENTS

• Testing with a real fire
• Publishing transformation of the fire
  • separating the thermal camera from the robot’s pincer
  • implementing depth sensing of the fire
• Mounting the Adroit on a wheeled mobile robot
  • Applying SLAM to the mobile robot
• Using a fire extinguisher with a flexible hose
• Picking up a fire extinguhsier using object detection