Leveraging Transfer Learning for Niche Sign System Recognition in VR Training with Limited Data

Abstract

Speech Supported by Gestures, in Dutch Spreken met On-dersteuning van Gebaren (SMOG), is a Belgian sign system that enhances verbal communication for individuals with communicative disabilities through specific gestures. Although effective, SMOG training is labor-intensive and typically requires one-on-one instruction. Virtual reality (VR) training can make the practice of SMOG gestures more scalable, playful, and cost-effective. For such a VR training to be effective , trainees should receive accurate and timely automated feedback on whether they perform the correct gestures. However, since SMOG and many other specialized motor skills are niche problems, there are no large annotated datasets to train machine learning models to perform this task from scratch, and collecting large amounts of data for such niche tasks is infeasible. We therefore propose using transfer learning to fine-tune pre-trained Mobile Video Networks (MoViNets) on a small dataset of RGB videos showing SMOG gestures. Through this workflow, we demonstrate recognition accuracies exceeding 99% using only two to five samples per gesture for training. This work therefore not only advances accessible SMOG training through autonomous VR practice but also establishes a highly data-and computation-efficient machine-learning framework for recognizing other niche sign systems or motor skills using limited amounts of training data.