Hand Localization Using YOLO on Depth Data

Abstract

The Human Interface Mate (HIM) of Arkite is an augmented reality device that assists production operators in real time. It uses 3D sensors and a projector to inform and guide operators in their work area during the production process. It can be improved by implementing real-time hand detection to check when an operator enters a predetermined area with his or her hand before proceeding to the next step in the production process. It is important to be able to detect a person’s hands with and without gloves equipped.

A literature study is first conducted, which leads to four promising solutions selected based on hardware requirements, performance and other criteria. Those possible solutions are YOLO, YOLO combined with OpenPose, Voxel-to-Voxel and Anchor-to-Joint. YOLO and OpenPose are pretrained for RGB images while Voxel-to-Voxel and Anchor-to-Joint require depth images. Work has been done to test each of the possible solutions after which it was decided to retrain YOLO to detect hands based only on depth images.

The retraining of YOLO is done on depth images to avoid any privacy concerns caused by RGB images. The ITOP and Arkite custom dataset is used for the training and testing. The results show a mean average precision (mAP) of 98.60% on the ITOP dataset and a good mAP of 13.59% on the Arkite dataset. Additional training is required to improve the mAP on the Arkite dataset.