Detecting Tool Keypoints with Synthetic Training Data

Abstract

In this paper an end-to-end technique is presented to create a deep learning model to detect 2D keypoint locations from RGB images. This approach is specifically applied to tools, but can be used on other objects as well. First, 3D models of similar objects are sourced form the internet to avoid the need for exact textured models of the target objects. It is shown in this paper that these exact 3D models are not needed. To avoid the high cost of manually creating a data set, an image generation technique is introduced that is specifically created to generate synthetic training data for deep learning models. Special care was taken when designing this method to ensure that models trained on this data generalize well to unseen, real world data. A neural network architecture, Intermediate Heatmap Model (IHM), is presented that can generate probability heatmaps to predict keypoint locations. This network is equipped with a type of intermediate supervision to improve the results on real world data, when trained on synthetic data. A number of other tricks are employed to ensure generalisation towards real world images. A dataset of real tool images is created to validate this approach. Validation shows that the proposed method works well on real world images. Comparison to two other techniques shows that this method outperforms them. Additionally, it is investigated which deep learning techniques, such as transfer learning and data augmentation, help towards generalization on real data.